Guaranteed Policy Performance in Reinforcement Learning Citation Voloshin, Cameron (2024) Guaranteed Policy Performance in Reinforcement Learning. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/n2fg-e554. https://resolver.caltech.edu/CaltechTHESIS:06062024-061120491 Abstract Decision-making is ubiquitous in everyday life. Increasingly, researchers are seeking answers on how to optimally solve sequential decision-making tasks. Thanks to recent availability of computation, advances in deep learning, and released open-sourced code, it has become easy to train a computational agent to make decisions in many domains. Nevertheless, in realistic scenarios where the consequences of failure are high, running a trained computational agent in the wild poses substantial risk. The goal of this thesis is to develop and advance techniques that guarantee a learned agent does what we expect it to do. The thesis tackles two central questions: 1) Given an agent, how can we predict if it will perform desirably? 2) Can we structure the learning process to guarantee desirable post-learning performance? On the former question, this thesis proposes multiple algorithms to evaluate such agents, finds factors that have high influence on the success of agent evaluation, and open-sources benchmarks for further development in the space. On the latter question, this thesis formulates desirable agent behavior as a constrained optimization with varying types of constraints depending on the structure afforded to the practitioner. Constraining the search space over the learning process ensures post-learning behaviors will, by definition, perform as desired. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Reinforcement Learning, Policy Learning, Off policy Evaluation Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Computing and Mathematical Sciences Thesis Availability: Public (worldwide access) Research Advisor(s): Yue, Yisong Thesis Committee: Wierman, Adam C. (chair) Yue, Yisong Bouman, Katherine L. Chaudhuri, Swarat Defense Date: 13 June 2023 Non-Caltech Author Email: clvoloshin (AT) gmail.com Record Number: CaltechTHESIS:06062024-061120491 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:06062024-061120491 DOI: 10.7907/n2fg-e554 Related URLs: URL URL Type Description https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/a5e00132373a7031000fd987a3c9f87b-Abstract-round1.html Publisher Article adapted for Chapter 3 https://proceedings.mlr.press/v130/voloshin21a.html Publisher Article adapted for Chapter 4 https://proceedings.mlr.press/v97/le19a.html Publisher Article adapted for Chapter 5 and 6 https://proceedings.neurips.cc/paper_files/paper/2022/hash/70b8505ac79e3e131756f793cd80eb8d-Abstract-Conference.html Publisher Article adapted for Chapter 7 https://proceedings.mlr.press/v202/voloshin23a.html Publisher Article adapted for Chapter 8 ORCID: Author ORCID Voloshin, Cameron 0009-0007-7725-6660 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 16508 Collection: CaltechTHESIS Deposited By: Cameron Voloshin Deposited On: 06 Jun 2024 23:04 Last Modified: 14 Jun 2024 21:18 Thesis Files PDF - Final Version See Usage Policy. 18MB Repository Staff Only: item control page

Guaranteed Policy Performance in Reinforcement Learning Thesis by Cameron Voloshin In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy California Institute of Technology Pasadena, California 2024 Defended June 13, 2023 ii © 2024 Cameron Voloshin ORCID: 0009-0007-7725-6660 cvoloshin.com All rights reserved iii ACKNOWLEDGEMENTS To my loving mother, Mila Voloshin. I am deeply grateful to my immediate family, Oleg, Etana Voloshin, and Danielle Bronner. You have been an incredible support system, and your unwavering belief in me has pushed me to strive for excellence. To all of the remarkable people that I have had the privilege of calling close friends: Thomas Anderson, Ada Campagna, Jessica Vautor, Joe Slote, Eitan Levin, Dima Burov, Terry Gdoutos, Jacob Gilman, Max Obolsky, Colin Miller, Jessica Wang, Melodie Kao, Rogelio Gomez, I want to express my heartfelt gratitude. You have been there for me through the highs and lows, and I have learned tremendously from each and every one of you. Your unique and special presence in my life have made this thesis possible. Without any doubt, I want to extend my sincerest appreciation to my advisor, Yisong Yue. Your guidance, support, insights, and encouragement before and throughout my PhD adventure have been invaluable. You have made my dreams possible many times over. My gratitude also goes out to the remaining members of my thesis committee, Adam Wierman, Katie Bouman, and Swarat Chaudhuri, for their insightful feedback and suggestions. I would like to thank my other collaborators, Hoang Le, Abhinav Verma, and Nan Jiang, as well as my Argo internship coworkers Andrew Hartnett and Matthew Hausknecht. Working with such amazing individuals has allowed me to learn and grow tremendously. It has also been a privilege to engage in discussions with my current and former Caltech colleagues, whose names are too numerous to mention individually. You all have contributed to creating a supportive and collaborative research environment. To all of you, thank you for being there for me and allowing me to lean on you. Your presence, guidance, and contributions have been instrumental in my journey, and I am truly grateful. iv ABSTRACT Decision-making is ubiquitous in everyday life. Increasingly, researchers are seeking answers on how to optimally solve sequential decision-making tasks. Thanks to recent availability of computation, advances in deep learning, and released open-sourced code, it has become easy to train a computational agent to make decisions in many domains. Nevertheless, in realistic scenarios where the consequences of failure are high, running a trained computational agent in the wild poses substantial risk. The goal of this thesis is to develop and advance techniques that guarantee a learned agent does what we expect it to do. The thesis tackles two central questions: 1) Given an agent, how can we predict if it will perform desirably? 2) Can we structure the learning process to guarantee desirable post-learning performance? On the former question, this thesis proposes multiple algorithms to evaluate such agents, finds factors that have high influence on the success of agent evaluation, and open-sources benchmarks for further development in the space. On the latter question, this thesis formulates desirable agent behavior as a constrained optimization with varying types of constraints depending on the structure afforded to the practitioner. Constraining the search space over the learning process ensures post-learning behaviors will, by definition, perform as desired. v PUBLISHED CONTENT AND CONTRIBUTIONS [1] Cameron Voloshin, Abhinav Verma, and Yisong Yue. “Eventual Discounting Temporal Logic Counterfactual Experience Replay”. In: International Conference on Machine Learning (ICML). 2023. CV participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and led in the writing of the manuscript. [2] Cameron Voloshin, Hoang Minh Le, Swarat Chaudhuri, and Yisong Yue. “Policy Optimization with Linear Temporal Logic Constraints”. In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview.net/forum?id=yZcPRIZEwOG. CV participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and led in the writing of the manuscript. [3] Cameron Voloshin, Nan Jiang, and Yisong Yue. “Minimax Model Learning”. In: International Conference on Artificial Intelligence and Statistics. PMLR. 2021, pp. 1612–1620. CV participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and led in the writing of the manuscript. [4] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. “Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning”. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Ed. by J. Vanschoren and S. Yeung. Vol. 1. 2021. url: https://openreview. net/forum?id=IsK8iKbL-I. CV participated in the conception of the project, formulated, implemented and analyzed the methods, prepared the data, conducted the experiments, and led in the writing of the manuscript. [5] Hoang M Le, Cameron Voloshin, and Yisong Yue. “Batch Policy Learning under Constraints”. In: International Conference on Machine Learning (ICML). 2019. CV participated in the formulation, implementation and experimentation in the project, and in the writing of the manuscript. vi TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Published Content and Contributions . . . . . . . . . . . . . . . . . . . Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii iv v v ix xi I 1 2 3 4 7 7 8 8 12 Introduction Chapter 1: Motivation and Goal . . . . . . . . . . . . . . . . . . . . . . 1.1 Guarantee from existing data: Off Policy Evaluation (OPE) . . . . 1.2 Guarantee from restructuring learning: Learning with Guarantees . Chapter 2: Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Preliminaries to Policy Learning, and Markov Decision Processes . . 2.2 Preliminaries to OPE . . . . . . . . . . . . . . . . . . . . . . . 2.3 Preliminaries to Linear Temporal Logic . . . . . . . . . . . . . . 2.4 Policy Learning With Guarantees . . . . . . . . . . . . . . . . . II Off Policy Evaluation Chapter 3: Caltech OPE Benchmark and Empirical Study . . . . . . . . . 3.1 Benchmarking Design & Methodology . . . . . . . . . . . . . . . 3.2 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . 3.3 Discussion and Future Directions . . . . . . . . . . . . . . . . . Chapter 4: Advances in Model Based OPE . . . . . . . . . . . . . . . . 4.1 Introduction to Model-Based OPE and OPO . . . . . . . . . . . 4.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Minimax Model Learning (MML) for OPE . . . . . . . . . . . . . 4.4 Off-Policy Optimization (OPO) . . . . . . . . . . . . . . . . . . 4.5 Scenarios & Considerations . . . . . . . . . . . . . . . . . . . . 4.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Other Related Work . . . . . . . . . . . . . . . . . . . . . . . 4.8 Discussion and Future Work . . . . . . . . . . . . . . . . . . . Chapter 5: Advances in Model Free OPE . . . . . . . . . . . . . . . . . 5.1 Fitted Q Evaluation (FQE) for Off Policy Evaluation . . . . . . . 5.2 Generalization Guarantee of FQE . . . . . . . . . . . . . . . . . 5.3 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . 14 15 17 24 29 32 32 33 34 39 41 44 47 48 49 49 49 51 vii III Policy Learning with Guarantees Chapter 6: Value-Based Guarantees . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . 6.3 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Other Related Work . . . . . . . . . . . . . . . . . . . . . . . 6.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 7: LTL-based Guarantees in Discrete Domains . . . . . . . . . . 7.1 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . 7.2 Background and Problem Formulation . . . . . . . . . . . . . . . 7.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 End-To-End Guarantees . . . . . . . . . . . . . . . . . . . . . . 7.5 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 8: LTL-based Guarantees in Continuous Domains . . . . . . . . . 8.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . 8.2 RL-Friendly Form: Eventual Discounting . . . . . . . . . . . . . 8.3 LTL Counterfactual Experience Replay . . . . . . . . . . . . . . 8.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV Appendix 52 53 53 55 58 61 64 67 68 69 70 71 73 79 80 81 82 84 85 86 90 93 96 97 120 Appendix A: Chapter 1 Appendix . . . . . . . . . . . . . . . . . . . . . 121 A.1 Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . 121 A.2 Ranking of Methods . . . . . . . . . . . . . . . . . . . . . . . 122 A.3 Supplementary Folklore Backup . . . . . . . . . . . . . . . . . . 125 A.4 Model Selection Guidelines . . . . . . . . . . . . . . . . . . . . 126 A.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 A.6 Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 A.7 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 133 A.8 Additional Supporting Figures . . . . . . . . . . . . . . . . . . . 138 Appendix B: Chapter 2 Appendix . . . . . . . . . . . . . . . . . . . . . 147 B.1 OPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 B.2 OPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 B.3 Additional theory . . . . . . . . . . . . . . . . . . . . . . . . . 157 B.4 Scenarios & Considerations . . . . . . . . . . . . . . . . . . . . 159 B.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Appendix C: Chapter 3 Appendix . . . . . . . . . . . . . . . . . . . . . 171 C.1 Preliminaries to Analysis of Fitted Q Evaluation (FQE) . . . . . . 171 C.2 Generalization Analysis of Fitted Q Evaluation . . . . . . . . . . . 174 viii Appendix D: Chapter 4 Appendix . . . . . . . . . . . . . . . . . . . . . 184 D.1 Equivalence between Regularization and Constraint Satisfaction . . 184 D.2 Convergence Proofs . . . . . . . . . . . . . . . . . . . . . . . . 187 D.3 End-to-end Generalization Analysis of Main Algorithm . . . . . . . 188 D.4 Preliminaries to Analysis of Fitted Q Iteration (FQI) . . . . . . . . 191 D.5 Finite-Sample Analysis of Fitted Q Iteration (FQI) . . . . . . . . . 193 D.6 Additional Instantiation of Meta-Algorithm (algorithm 4) . . . . . 199 D.7 Additional Experimental Details . . . . . . . . . . . . . . . . . . 201 Appendix E: Chapter 5 Appendix . . . . . . . . . . . . . . . . . . . . . 205 E.1 Notation and Overview . . . . . . . . . . . . . . . . . . . . . . 205 E.2 Analysis: Statements with Proof . . . . . . . . . . . . . . . . . 208 E.3 Conjecture on Sample Complexity . . . . . . . . . . . . . . . . . 225 E.4 Additional Algorithms . . . . . . . . . . . . . . . . . . . . . . . 226 E.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Appendix F: Chapter 6 Appendix . . . . . . . . . . . . . . . . . . . . . 237 F.1 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 F.2 Constructing feasible trajectories for policy gradient during rollout . 240 ix LIST OF ILLUSTRATIONS Number Page 2.1 LTL-Based Guarantees, Examples . . . . . . . . . . . . . . . . . 9 3.1 Example of an OPE factor . . . . . . . . . . . . . . . . . . . . 19 3.2 OPE Method Comparison . . . . . . . . . . . . . . . . . . . . . 24 3.3 OPE Method Decision Tree . . . . . . . . . . . . . . . . . . . . 25 3.4 OPE Method Comparison . . . . . . . . . . . . . . . . . . . . . 26 3.5 OPE Method Comparison . . . . . . . . . . . . . . . . . . . . . 27 4.1 Visual Representation of MML Loss . . . . . . . . . . . . . . . . 36 4.2 MML Experiment Results for LQR . . . . . . . . . . . . . . . . . 45 4.3 MML Experiment Results for Cartpole . . . . . . . . . . . . . . . 46 4.4 MML Experiment Results for Inverted Pendulum . . . . . . . . . . 47 6.1 Value-Based Guarantees Experiment for FrozenLake . . . . . . . . 65 6.2 Value-Based Guarantees Experiment for CarRacing . . . . . . . . . 65 7.1 Motivating Examples for LTL-Based Guarantees . . . . . . . . . . 70 7.2 Product MDP Diagrams . . . . . . . . . . . . . . . . . . . . . 73 7.3 LTL-Based Guarantees, Discrete - Results . . . . . . . . . . . . . 80 8.1 LTL-Based Guarantees, Examples . . . . . . . . . . . . . . . . . 85 8.2 LTL-Based Guarantees, Continuous - Results . . . . . . . . . . . 93 A.1 Environment Illustration: Graph . . . . . . . . . . . . . . . . . . 130 A.2 Environment Illustration: Graph-Mountain Car . . . . . . . . . . . 130 A.3 Environment Illustration: Mountain Car . . . . . . . . . . . . . . 130 A.4 Environment Illustration: Enduro . . . . . . . . . . . . . . . . . 130 A.5 Environment Illustration: Graph-POMDP . . . . . . . . . . . . . 130 A.6 Environment Illustration: Gridworld . . . . . . . . . . . . . . . . 130 A.7 Experiment Hyperparameters by Model and Environment . . . . . 136 A.8 Hyperparameters for each model by Environment, Cont. . . . . . . 137 A.9 Enduro DM vs IPS. . . . . . . . . . . . . . . . . . . . . . . . . 138 A.10 MC comparison - Function approx . . . . . . . . . . . . . . . . . 138 A.11 Enduro DM vs HM . . . . . . . . . . . . . . . . . . . . . . . . 139 A.12 Short vs Long Horizon comparison - Graph . . . . . . . . . . . . 139 A.13 Short vs Long Horizon comparison - Graph . . . . . . . . . . . . 139 A.14 Short vs Long Horizon comparison - DM vs DR . . . . . . . . . . 139 x A.15 A.16 A.17 A.18 A.19 A.20 A.21 A.22 A.23 A.24 A.25 A.26 A.27 A.28 A.29 A.30 A.31 A.32 A.33 A.34 A.35 A.36 A.37 A.38 A.39 B.1 D.1 D.2 E.1 E.2 E.3 E.4 Comparison in a low data regime . . . . . . . . . . . . . . . . . 140 Comparison with dense vs sparse rewards . . . . . . . . . . . . . 140 Comparison with exact vs estimated πb . . . . . . . . . . . . . . 140 Comparison with exact vs estimated πb . . . . . . . . . . . . . . 140 Hybrid Method comparison . . . . . . . . . . . . . . . . . . . . 141 Hybrid Method comparison . . . . . . . . . . . . . . . . . . . . 141 Hybrid Method comparison . . . . . . . . . . . . . . . . . . . . 141 Hybrid Method comparison . . . . . . . . . . . . . . . . . . . . 141 Hybrid Method comparison . . . . . . . . . . . . . . . . . . . . 142 Class comparison with unknown πb . . . . . . . . . . . . . . . . 142 Class comparison with unknown πb . . . . . . . . . . . . . . . . 142 Class comparison with unknown πb . . . . . . . . . . . . . . . . 142 AM Direct vs Hybrid comparison for AM. (Gridworld) . . . . . . . 143 FQE Direct vs Hybrid comparison . . . . . . . . . . . . . . . . . 143 MRDR Direct vs Hybrid comparison . . . . . . . . . . . . . . . . 143 Q-Reg Direct vs Hybrid comparison . . . . . . . . . . . . . . . . 144 Qπ (λ) Direct vs Hybrid comparison . . . . . . . . . . . . . . . . 144 Retrace(λ) Direct vs Hybrid comparison . . . . . . . . . . . . . . 144 Tree-Backup Direct vs Hybrid comparison . . . . . . . . . . . . . 145 DR comparison with varying DM . . . . . . . . . . . . . . . . . 145 WDR comparison with varying DM . . . . . . . . . . . . . . . . 145 MAGIC comparison with varying DM . . . . . . . . . . . . . . . 145 DR comparison with varying DM . . . . . . . . . . . . . . . . . 145 WDR comparison with varying DM . . . . . . . . . . . . . . . . 146 MAGIC comparison with varying DM . . . . . . . . . . . . . . . 146 Effect of increased Function Class size on MML . . . . . . . . . . 170 Environment Illustrations: FrozenLake and CarRacing . . . . . . . 201 Experiment Results for LSPI and Value-Based Constraints . . . . . 203 Demonstration of Blocking Issue of LCP . . . . . . . . . . . . . . 227 Environment Illustrations: Pacman, Mountain Car . . . . . . . . . 231 Additional Results for LTL-Based Constraints, Discrete . . . . . . . 234 Effect on policy for varying LTL objectives . . . . . . . . . . . . . 235 xi LIST OF TABLES Number Page 3.1 Benchmark environment characteristics . . . . . . . . . . . . . 20 7.1 Operators and Parameters for Value Iteration . . . . . . . . . . 78 A.1 Summary of Near-Top Frequency . . . . . . . . . . . . . . . . 122 A.2 OPE - Short Horizon, Small Policy Mismatch . . . . . . . . . . 123 A.3 OPE - Short Horizon, Large Policy Mismatch . . . . . . . . . . 123 A.4 OPE - Long Horizon, Small Policy Mismatch . . . . . . . . . . 123 A.5 OPE - Long Horizon, Large Policy Mismatch, Deterministic Env 123 A.6 OPE - Long Horizon, Large Policy Mismatch, Stochastic Env . . 124 A.7 OPE - Insufficient Representation . . . . . . . . . . . . . . . . 124 A.8 OPE - Sufficient Representation, Poor πb Estimation . . . . . . 124 A.9 OPE - Sufficient Representation, Good πb Estimation . . . . . . 124 A.10 Folklore Backup - Baseline . . . . . . . . . . . . . . . . . . . . 125 A.11 Folklore Backup - Increasing Horizon . . . . . . . . . . . . . . 125 A.12 Folklore Backup - Increasing Policy Mismatch . . . . . . . . . . 125 A.13 OPE Model Selection Guidelines . . . . . . . . . . . . . . . . . 126 A.14 Model Selection Guidelines Cont. . . . . . . . . . . . . . . . . 126 A.15 IPS Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 A.16 Full COBS Environment Parameters . . . . . . . . . . . . . . . 133 A.17 Experiment Parameters - Graph . . . . . . . . . . . . . . . . . 134 A.18 Experiment Parameters - Graph-POMDP . . . . . . . . . . . . 134 A.19 Experiment Parameters - Gridworld . . . . . . . . . . . . . . . 134 A.20 Experiment Parameters - Pixel Gridworld . . . . . . . . . . . . 134 A.21 Experiment Parameters - Graph Mountain Car . . . . . . . . . 134 A.22 Experiment Parameters - Mountain Car . . . . . . . . . . . . . 134 A.23 Experiment Parameters - Pixel Mountain Car . . . . . . . . . . 135 A.24 Experiment Parameters - Enduro . . . . . . . . . . . . . . . . 135 E.1 Description of Policies and Probabilities . . . . . . . . . . . . . 206 E.2 Description of Gains . . . . . . . . . . . . . . . . . . . . . . . 206 E.3 Description of Value Functions . . . . . . . . . . . . . . . . . . 207 E.4 LTL-Based Guarantees, Discrete - Experiment Hyperparameters 233 F.1 LTL-Based Guarantees, Continuous - Environment Details . . . 237 F.2 LTL-Based Guarantees, Continuous - Q-learning Hyperparam . 238 F.3 LTL-Based Guarantees, Continuous - PPO Hyperparam . . . . 239 Part I INTRODUCTION 2 Chapter 1 MOTIVATION AND GOAL This chapter motivates and presents the goal and developments of this thesis. Simply put, solving a sequential decision-making task involves finding a sequence of actions over time that achieve a desired outcome. It is easy to appreciate the wide range of tasks to which this concept applies, such as traffic routing, power system management, recommendation systems, financial trading, motion planning in autonomous vehicles and robotics, games like chess, and even simple tasks like making a sandwich. The list is abundant. Beyond some basic systems, classical solutions have become impractical due to the extensive search space they require. Increasingly, researchers and practitioners have leaned on learning-based approaches, fueled by superhuman levels of computation, leveraging success in deep learning including computer vision [112, 49], and increasingly language models [213]. These powerful deep learning advances have contributed to dramatic successes in sequential decision domains, most notably in games [144, 185] and recommendation [127]. However, many decision-making domains involve substantial risks to human life, hardware, and resources. This raises a fundamental question when deploying “learned agents” capable of interacting with the world: How can we be sure that the agent will succeed? (Performance Guarantee) As of the writing of this thesis, providing a satisfying answer to this question remains elusive, limiting the applicability of learning-based approaches to systems where repeated failure is tolerable. In the following sections, we present our attempt to address this fundamental question. Our approach consists of two key steps: 1) We will massage any existing data to extract meaningful signal on our agent’s expected performance. 3 2) We will re-frame the learning process to proactively ensure satisfactory performance of the agent. 1.1 Guarantee from existing data: Off Policy Evaluation (OPE) A key approach to providing a performance guarantee for a learned agent, particularly when the cost of deployment is high, involves leveraging existing data. This data may include human demonstrations, or limited deployments of classical or conservative agents. The problem of estimating the performance of a new agent using pre-collected historical data generated by other agents is known as the Off-Policy Policy Evaluation (OPE) problem. In practice, the dataset contains actions chosen by a different agent from the one being evaluated. Had we run the new agent in the world we may have observed different actions and outcomes than the ones available in the dataset. Accounting for this distribution shift poses the fundamental challenge in OPE. Ignoring this shift can result in poor estimation of agent performance. Given its importance, the research community actively advances OPE techniques, both for the bandit [54, 24, 196, 219, 128, 138] and reinforcement learning settings [97, 54, 60, 133, 223, 209, 152, 225, 44]. These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations [127, 220, 24, 17], and the increasing importance of off-policy learning and counterfactual reasoning [47, 203, 149, 123, 135, 155]. OPE is also similar to the dynamic treatment regime problem in the causal inference literature [151]. In Part II, our work focuses on OPE. Chapter 3 surveys and benchmarks the existing techniques for OPE, teasing out some of the key factors that have high influence on method performance. We summarize guidelines for method selection depending on the circumstance and open source an extensive package for new OPE experimentation and development. We also outline further directions remaining in OPE development. To the best of our knowledge, this is the first comprehensive study on OPE. Chapter 4 restricts its attention to model-based OPE. The chapter argues that current model-based approaches for OPE have an incomplete learning objective and do not properly correct for distribution shift. We develop a novel method known as Minimax Model Learning (MML), provide theoretical guarantees 4 on its predictive performance, and demonstrate empirical improvement over previous methods. We also show that this idea can, additionally, be extended to do learning, which we will get to in the next section. Similarly, Chapter 5 attends to model-free OPE. Here we develop Fitted Q Evaluation (FQE), a simple method for OPE and derive guarantees on its predictive performance. It is demonstrably more reliable than previous model-free methods and significantly easier to implement. 1.2 Guarantee from restructuring learning: Learning with Guarantees A policy serves as the decision system that guides an agent’s actions within the world. The interaction protocol follows a sequence of steps: the world reveals a context, the agent selects an action based on that context using its policy, the agent executes the action and experiences a cost, the world is influenced by the action, and the next context is revealed, and so on. Typically, a policy maps previous contexts to a distribution of possible actions. Common assumptions include the policy depending only on the last context (Markovian) and the environment providing all relevant information (fully observable). This process is formalized as a Markov Decision Process (MDP) [166] and is foundational to the reinforcement learning framework [190, 21]. At a high level, the primary objective of the interaction loop is to find sequences of actions that minimize the agent’s accumulated cost, leading to optimal solutions. By achieving this, the agent is expected to demonstrate desired behaviors and successfully accomplish whatever task is implied within the costs. Indeed, the underlying premise, known as the “reward hypothesis” [194, 186], posits that a scalar cost function fully captures the task specification. However, designing a single appropriate scalar cost function can be excessively challenging, particularly in complex real-world systems like self-driving cars. These tasks often involve numerous constraints, like safety, reliability, and comfort, in addition to a main objective like reaching a destination. In the context of the reward hypothesis, these constraints need to be integrated into the scalar cost function. This blending of the main objective and real-world constraints introduces complexity in both guaranteeing and validating the success of the agent’s behavior. In fact, designing a cost (reward) function can be so hard that there is an entire effort to investigate learning a cost function [79, 63, 233, 188]. Finding 5 the correct cost function can be as hard as solving the problem itself [154]. Recent theoretical evidence suggests that certain tasks are simply not reducible to scalar costs [2]. Nevertheless, to date, almost all theoretical understanding of RL is focused on this scalar cost minimization setting (e.g., [207, 100, 101, 197, 160, 14, 72, 45, 15, 6, 7, 126, 167, 168]). In practice, one circumvents these challenges using heuristics such as adding “breadcrumbs”, known as cost-shaping [187], to guide the agent to qualitatively behave the way the practitioner intended. However, such heuristics can lead to catastrophic failures in which the learning agent ends up exploiting the cost function in an unanticipated way [171, 206, 93, 227, 154]. Additionally, the process of cost-shaping often requires iterative refinement, as practitioners are unlikely to achieve the desired behavior on their first attempt. Consequently, this iterative process is only viable when the cost of failure is relatively low. As far as guarantees are concerned, a cost-shaped function has lost precision in specifying the desired behavior. To address these limitations and pitfalls, Part III of this thesis focuses on reconfiguring the learning problem without relying on the reward hypothesis. Instead, we explore two alternative approaches: 1) Value-Based Guarantees 2) Linear Temporal Logic (LTL)-Based Guarantees Learning with Value Based Guarantees In this approach, the domain expert decomposes the task into multiple simultaneous constraints, each associated with a cost function and a corresponding value function (cumulative cost). The objective is to keep each value function below its respective threshold, which may be determined by prior domain knowledge or historical data. Chapter 6 presents an approach that allows the imposition of value-based constraints in a counterfactual manner when historical data already exists, without the need for further agent-world interaction. We provide end-to-end finite-sample performance guarantees, leveraging Off-Policy Policy Evaluation (Part II) to reason about constraint satisfaction. 6 Learning with LTL-Based Guarantees In this setting, the domain expert specifies the task using the language of Linear Temporal Logic (LTL). Unlike a scalar cost function, LTL enables precise description of a task and further enables transparent blending of any additional constraints into a single specification. When desired, a cost function can be defined to discern the “best” LTL-satisfying policy, based on factors like time, energy, or effort. Chapter 7 introduces an approach to finding (cost-optimal) LTL-satisfying policies in discrete context systems with access to a generative model of the world. Strong guarantees are provided with minimal assumptions. We develop several theoretical tools for our analyses and empirically validate the strength of our method. In Chapter 8, we address the scenario where a generative model is not available, and an LTL-satisfying policy needs to be learned through online interaction with the environment. We develop a proxy value function and a technique called LTLguided counterfactual experience replay (LCER) to overcome the challenges of sparse learning signals, allowing us to take advantage of specific structure afforded to us by LTL over traditional cost-functions. Empirical evaluations demonstrate the performance gains of LCER with various reinforcement learning algorithms. 7 Chapter 2 FOUNDATIONS This chapter introduces the reader to the preliminaries underlying this thesis. 2.1 Preliminaries to Policy Learning, and Markov Decision Processes As discussed in the previous chapter, sequential decision making involves iterated interaction between agent and world. Formally, this is abstracted to discounted MDP framework specified by a tuple (S, A, P, R, γ). Here, S is the state (context) space, A is the action space, P : S × A → ∆(S) is the transition function of the world, R : S × A → ∆([−R̄, R̄]) is the reward function, and γ ∈ [0, 1) is the discount factor. It may sometimes be convenient to consider C : S × A → ∆([−C̄, C̄]), a cost function instead of a reward function. Here, C̄, R̄ ∈ R. Let X ≡ S × A. Given an MDP, a (stochastic) policy π : S → ∆(A) and a starting state distribution d0 ∈ ∆(S) together determine a distribution over trajectories of the form s0 , a0 , r0 , s1 , a1 , r1 , . . . , where s0 ∼ d0 , at ∼ π(st ), rt ∼ R(st , at ), and st+1 ∼ P (st , at ) for t ≥ 0. The standard optimization objective for policy learning is to optimize the value function (or cumulative reward) defined as:  ∞ X V P (π) ≡ E  γ i ri i=0  s0 ∼d0 ai ∼π(·|si )  si+1 ∼P (·|si ,ai ) , ri ∼R(·|si ,ai ) (2.1) over all π in some policy class Π. It will occasionally be convenient to keep track of the action-value function defined as: # " Qπ (s, a) ≡ E s0 =s,a0 =a a ∼π(·|s ) γ ri si+1i∼P (·|sii ,ai ) ri ∼R(·|si ,ai ) i=0 ∞ X i , Qπ (s, π(s)) ≡ Ea∼π(s) [Qπ (s, a)] . h i It is easy to see that V P (π) = Es∼d0 [Qπ (s, π(s))] = Es∼d0 Ea∼π(s)) [Qπ (s, a)] . 8 2.2 Preliminaries to OPE Since policies are selected by optimizing Eq (2.1), we can evaluate it using the same metric: V P (π) ≡ J(π, P ) ≡ Es∼d0 [VπP (s)], (2.2) where we can recursively write V using the Bellman Equation, VπP (s) ≡ Ea∼π(·|s) [Er∼R(·|s,a) [r] + γEs̃∼P (·|s,a) [VπP (s̃)]]. (2.3) A useful equivalent measure of performance is: J(π, P ) = E(s,a)∼dPπ,γ [Er∼R(·|s,a) [r]], (2.4) where dPπ,γ (s, a) ≡ t=0 γ t dPπ,t (s, a) is the (discounted) distribution of stateaction pairs induced by running π in P and dPπ,t ∈ ∆(X ) is the distribution of (st , at ) induced by running π under P . The first term in dPπ,γ is dPπ,0 = d0 . dPπ,t has a recursive definition that we use in Section 4.3: P∞ dPπ,t (s, a) = Z dPπ,t−1 (s̃, ã)P (s|s̃, ã)π(a|s)dν(s̃, ã), (2.5) where ν is the Lebesgue measure. In the batch learning setting, we are given a dataset D = {(si , ai , s′i )}ni=1 , where si ∼ dπb (s), ai ∼ πb , and s′i ∼ P (·|si , ai ), where πb is some behavior policy that collects the data. For convenience, we write (s, a, s′ ) ∼ Dπb P , where Dπb (s, a) = dπb (s)πb (a|s). The OPE objective is to estimate: J(π, P ∗ ) ≡ V P∗  ∞ X (π) ≡ E  γ i ri i=0  s0 ∼d0 ai ∼π(·|si )  si+1 ∼P ∗ (·|si ,ai ) , ri ∼R(·|si ,ai ) (2.6) the performance of an evaluation policy π in the true environment P ∗ , using only logging data D with samples from Dπb P ∗ . Solving this objective is difficult because the actions in our dataset were chosen with πb rather than π. Thus, any π ̸= πb potentially induces a “shifted” state-action distribution Dπ ̸= Dπb , and ignoring this shift can lead to poor estimation. OPE is sometimes considered in the episodic RL setting. In this situation, D = {τ i }N i=1 , a set of N trajectories (or episodes), where i indexes over trajectories, and τ i = (si0 , ai0 , r0i , . . . , siT −1 , aiT −1 , rTi −1 ) and T may be fixed or a random variable. 2.3 Preliminaries to Linear Temporal Logic We give the necessary background and examples to understand Linear Temporal Logic. An atomic proposition is a variable that takes on a truth value. An 9 alphabet over a set of atomic propositions AP is given by Σ = 2AP . For example, if AP = {x, y} then Σ = {{}, {x}, {y}, {x, y}}. ∆(A) represents the set of probability distributions over a set A. Running Example ⊥ 0  1 1 ¬y ⊥ y 2 y ¬y 0 ¬r r b b 2 ¬r 3 b ⊥ Figure 2.1: Examples. First: Illustration of the Flatworld environment. The agent is a green dot and there are 3 zones: yellow, blue and red. Second: ∗ LDBA B for “F Gy”. S B = {1}, denoted by green circle. The initial state is ∗ b−1 = 0. Third: LDBA B for “GF (y & XF r) & G¬b”. S B = {1}, denoted by green circle. The initial state is b−1 = 1. We will be using the environment illustrated in Figure 2.1 (First) as a running example. The agent is given by a green dot and there are three circular regions in the environment colored yellow, blue, and red. The AP are given by {y, b, r}, referring to the respective colored zones. Elements of Σ indicate which zone(s) the agent is in. MDPs with Labelled State Spaces We extend an MDP from Section 2.1 to a labelled Markov Decision Process M (MDP) framework given by the tuple M = (S M , AM , P M , dM 0 , γ, L ) consisting of a state space S M , an action space AM , an unknown transition function M P M : S M × AM → ∆(S M ), an initial state distribution dM 0 ∈ ∆(S ), and a labelling function LM : S M → Σ. Let AM (s) to be the set of available actions in state s. Notice, unlike traditional MDPs, M has a labeling function LM which returns the atomic propositions that are true in that state. For example, in Figure 2.1 (First), when the agent enters a state s ∈ S M such that it is both in the yellow and blue zone then LM (s) = {y, b}. 10 Linear Temporal Logic (LTL) Here we give a basic introduction to LTL. For a more comprehensive overview, see Baier and Katoen [16]. Definition 2.3.1 (LTL Specification, φ). An LTL specification φ is the entire description of the task, constructed from a composition of atomic propositions with logical connectives: not (¬), and (&), and implies (→); and temporal operators: next (X), repeatedly/always/globally (G), eventually (F ), and until (U ). Examples. For AP = {x, y}, some basic task specifications include safety (G¬x), reachability (F x), stability (F Gx), response (x → F y), and progress (x & XF y). Consider again the environment in Figure 2.1 (First) where AP = {y, r, b}. If the task is to eventually reach the yellow zone and stay there (known as stabilization) then we write φ = F Gy. Or, if we would like the agent to infinitely loop between the yellow and red zone while avoiding the blue zone then φ = GF (y & XF r) & G¬b, a combination of safety, reachability, and progress. LTL Satisfaction LTL has recursive semantics defining the meaning for logical connective satisfaction. Without loss of generality, we will be using a specialized automaton, an LDBA Bφ [184], defined below to keep track of the progression of φ satisfaction. More details for constructing LDBAs are in [80, 16, 111]. We drop φ from Bφ for brevity. Definition 2.3.2. (Limit Deterministic Büchi Automaton, LDBA [184]) An ∗ LDBA is a tuple B = (S B , Σ ∪ AB , P B , S B , bB−1 ) consisting of (i) a finite set of states S B , (ii) a finite alphabet Σ = 2AP , AB is a set of indexed jump transitions (iii) a transition function P B : S B × (Σ ∪ AB ) → S B , (iv) accepting ∗ states S B ⊆ S B , and (v) initial state bB−1 . There exists a mutually exclusive ∗ B B B partitioning of S B = SD ∪ SNB such that S B ⊆ SD , and for b ∈ SD , a ∈ Σ then B B P (b, a) ⊆ SD , closed. AB (b) is only (possibly) non-empty for b ∈ SNB and B allows B to transition to SD without reading an AP. A path ϱ = (b0 , b1 , . . .) is a sequence of states in B reached through successive transitions under P B . 11 Definition 2.3.3. (B accepts) B accepts a path ϱ if there exists some state ∗ b ∈ S B in the path that is visited infinitely often. Examples. Consider again the environment in Figure 2.1 (First) where AP = {y, r, b}. If we would like to make an LDBA for φ = F Gy (reach and stabilize at y) then we would get the state machine seen in Figure 2.1 (Second). In this state machine, the agent starts at state 0. The accepting set is given ∗ by S B = {1}. The transition between state 0 and state 1 is what is formally referred to as a jump transition: AB (0) = {ϵ} while AB (·) = ∅ otherwise. Whenever the agent is in state 0 of the LDBA, there is a choice of whether to stay at state 0 or transition immediately to state 1. This choice amounts to the agent believing that it has satisfied the “eventually” part of the LTL specification. When the agent takes this jump, then it must thereafter satisfy y to stay in state 1. The agent gets the decision of when it believes it is capable of satisfying y thereafter. When the agent takes the jump, if it fails to stay in y, it immediately transitions to the sink, denoted state 2. The LDBA accepts when the state 1 is reached infinitely often, meaning the agent satisfies “always y” eventually, as desired. Another example, this time without jump transitions, would be for φ = GF (y & XF r) & G¬b (oscillate between y and r forever while avoiding b). The LDBA can be seen in Figure 2.1 (Third). In this state machine, the agent ∗ starts at state 1 and the accepting set is given by S B = {1}. To make a loop back to state 1, the agent must visit both r and y. Doing so infinitely often satisfies the LDBA condition and therefore the specification. If at any point b is encountered then the agent transitions to the sink, denoted state 3. We first introduce slightly more notation. Let Z = S M × S B . Let Π : Z × A → ∆([0, 1]) be a (stochastic) policy class over the product space of the MDP and the LDBA (defined below), where A((s, b)) = AM (s) ∪ AB (b), to account for jump transitions in B. Synchronizing the MDP with the LDBA. For any (s, b) ∈ Z, a policy π ∈ Π is able to select an action in AM (s) or an action in AB (b), if available. We can therefore generate a trajectory as the sequence τ = (s0 , b0 , a0 , s1 , b0 , a1 , . . .) 12 under a new probabilistic transition relation given by P (s′ , b′ |s, b, a) =    P M (s, a, s′ )    1,     0, a ∈ AM (s), b′ ∈ P B (b, L(s′ )) a ∈ AB (b), b′ ∈ P B (b, a), s = s′ . (2.7) otherwise Let the LDBA projection of τ be the subsequence τB = (b0 , b1 , . . .). Elements of τB can be thought of as tracking an agent’s LTL specification satisfaction: Definition 2.3.4 (Run Satisfaction, τ |= φ). We say a trajectory satisfies φ if ∗ B accepts τB , which happens if ∃b ∈ τB infinitely often with b ∈ S B . Let TPπ = Ez∼dM [TPπ (z)] be the distribution over all possible trajectories 0 ×{b−1 } P starting from any initial state z ∈ dM 0 × {b−1 } where Tπ (z) is the (conditional) distribution over all possible trajectories starting from z ∈ Z generated by π under relation P (given in (2.7)). The probability of LTL satisfaction results from counting how many of the trajectories satisfy the LTL specification: Definition 2.3.5 (State Satisfaction, z |= φ). Pπ [z |= φ] = Eτ ∼TPπ (z) [1{τ |=φ} ] = Eτ ∼TPπ [1{τ |=φ} |z0 = z]. Definition 2.3.6 (Policy Satisfaction, π |= φ). P[π |= φ] = Eτ ∼TPπ [1{τ |=φ} ] where 1X is the indicator for X. 2.4 Policy Learning With Guarantees Optimizing the standard objective in Eq (2.1) requires that the whole task be encoded in the reward/cost function given by R or C. Instead, we give a brief introduction to the problems we will be solving instead, which give more nuanced guarantee over policy performance. Value-Based Guarantees In the case there are real-world constraints which have their own individual cost functions to keep under control we may take an objective of the form: arg min π∈Π s.t. V (π) G1 (π) ≤ τ1 , . . . , Gk (π) ≤ τk , (Value-Based Guarantees) t where the k constraint value functions are Gπi ≡ Es∼d0 [ ∞ t=0 γ gi (st , at )|s0 = s]. Each gi : X → R are known constraint cost functions and τi represents the allowable threshold. For examples of problems of this form, see Chapter 6. P 13 LTL-Based Guarantees Similarly, we can instead optimize over probability optimal policies in the form: arg max P [π |= φ] π∈Π (LTL-Based Guarantees) to find the policies that best satisfy our task, encoded in an LTL formula φ. For examples of problems of this form, see Chapter 8. Further, we could write this as a constrained problem in the form: arg max V (π) (LTL-Based Guarantees, with auxiliary cost) π∈Π s.t. π ∈ {arg max P [π ′ |= φ]}. ′ π ∈Π For examples of problems of this form, see Chapter 7. Remark 2.4.1. The policy class Π is different between the Value-Based Guarantees and LTL-Based Guarantees. In the former, Π : S → ∆(A) are functions from MDP states to MDP actions. In the latter, Π : (S M ×S B ) → ∆(AM ∪AB ) are functions from states to actions in the product MDP given by synchronizing a labelled MDP with an LDBA. Part II OFF POLICY EVALUATION 15 Chapter 3 CALTECH OPE BENCHMARK AND EMPIRICAL STUDY In this chapter, we develop an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We also distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark. We present the Caltech OPE Benchmarking Suite (COBS), which benchmarks OPE techniques via experimental designs that give thorough considerations to factors that influence performance. The reality of method performance, as we will discuss, is nuanced and comparison among different estimators is tricky without pushing the experimental conditions along various dimensions. Our philosophy and contributions can be summarized as follows: • We establish a benchmarking methodology that considers key factors that influence OPE performance, and design a set of domains and experiments to systematically expose these factors. The proposed experimental domains are complementary to continuous control domains from recent offline RL benchmarks [69, 68]. We differ from these recent benchmarks in two important ways: 1) COBS allows researchers fine-grained control over experimental design, other than just access to a pre-collected dataset. The offline data can be generated “on-the-fly” based on experimental criteria, e.g., the divergence between behavior and target policies. 16 2) We offer significant diversity in experimental domains, covering a wide range of dimensionality and stochasticity. Together, the goal of this greater level of access is to enable a deeper look at when and why certain methods work well. • As a case study, we select a representative set of established OPE baseline methods, and test them systematically. We further show how to distill the empirical findings into key insights to guide practitioners and inform researchers on directions for future exploration. • COBS is an extensive software package that can interface with new environments and methods to run new OPE experiments at scale.1 Given the fast-changing nature of this active area of research, our package is designed to accommodate the rapidly growing body of OPE estimators. COBS is already actively used by multiple research groups to benchmark new algorithms. Prior Work. Empirical benchmarks have long contributed to the scientific understanding, advancement, and validation of machine learning techniques [37, 35, 36, 177, 53, 49]. Recently, many have called for careful examination of empirical findings of contemporary deep learning and deep reinforcement learning efforts [89, 136]. As OPE is central to real-world applications of reinforcement learning, proper benchmarking is critical to ensure in-depth understanding and accelerate progress. While many recent methods are built on sound mathematical principles, a notable gap in the current literature is a standard for benchmarking empirical studies, with perhaps a notable exception from the recent DOPE [69] and D4RL benchmarks [68]. Compared to prior complementary work on OPE evaluation for reinforcement learning [68, 69], our benchmark offers two main advantages. First, we focus on maximizing reproducibility and nuanced experimental control with minimal effort, covering data generation and fine-grained control over factors such as relative “distance” between the offline data distribution and the distribution induced by evaluation policies. Second, we study a diverse set of environments, spanning range of desiderata such as stochastic-vs-deterministic and different representations for the same underlying environment. Together, these attributes enable our benchmarking suite to conduct systematic analyses of the method 1 https://github.com/clvoloshin/COBS 17 performance under different scenarios, and provide a holistic summary of the challenges one may encounter in different scenarios. Background & Notation. Recall from the preliminaries of Section 2.2, we are given an evaluation policy πe , the OPE problem is to estimate the hP i T −1 t value V (πe ) = Es∼d0 γ r |s = s , with at ∼ πe (·|st ), st+1 ∼ P (·|st , at ), t 0 t=0 rt ∼ R(st , at ), and d0 ⊆ S is the initial state distribution. For notational convenience in this chapter, we assume a fixed episode length of T . 3.1 Benchmarking Design & Methodology Design Philosophy The design philosophy of the Caltech OPE Benchmarking Suite (COBS) starts with the most prominent decision factors that can make OPE difficult. These factors come from both the existing literature and our own experimental study, which we will further discuss. We then seek to design experimental conditions that cover a diverse range of these factors. As a sub-problem within the broader reinforcement learning problem class, OPE experiments in existing literature gravitate towards commonly used RL domains. Unsurprisingly, the most common experiments belong to the Mujoco group of deterministic continuous control tasks [204], or discrete domains that operate via OpenAI Gym interface [31]. For OPE, high-dimensional domains such as Atari [20] appear less often, but are also natural candidates for OPE testing. We selectively pick from these domains as well as design new domains, with the goal of establishing refined control over the decision factors. Design Factors. We consider several domain characteristics that are often major factors in performance of OPE methods: 1) Horizon length. Long horizons can lead to catastrophic failure in some OPE methods due to an exponential blow-up in some of their components [128, 97, 133]. 2) Reward sparsity. Sparse rewards represent a difficult credit assignment problem in RL. This factor is often not emphasized in OPE, and arguably goes hand-in-hand with horizon length.2 3) Environment stochasticity. Popular RL domains such as Mujoco and Atari are mostly deterministic. This is a fundamental limitation in many existing empirical studies since many theoretical challenges to RL only surface in 2 Considered in isolation, long horizons may not be an issue if the reward signal is dense. 18 a stochastic setting. A concrete example is the famous double sampling problem [46], which is not applicable in many contemporary RL benchmarks. 4) Unknown behavior policy. This is related to the source of the collected data. The data may come from one or a more policies which may not be known. For example, existing dataset benchmarks, such as D4RL [68] can be considered to come from an unknown behavior policy. Some methods will require behavior policy estimation, thus introducing some bias. 5) Policy and distribution mismatch. The relative difference between the evaluation and behavior policy can play a critical role in the performance of many OPE methods. This difference induces a distribution mismatch between the dataset D and the dataset that would have been produced had we run the evaluation policy. Performing out-of-distribution estimation is a key challenge for robust OPE. We focus on providing a systematic way to stress test OPE methods under this mismatch, which we accomplish by offering a control knob for flexible data generation to induce various degrees of mismatch. 6) Model misspecification. Model misspecification refers to the insufficient representation power of the function class used to approximate different objects of interest, whether the transition dynamics, value functions, or state distribution density ratio. In realistic applications, it is reasonable to expect at least some degree of misspecification. We study the effect of misspecification via two controlled scenarios: (i) we start with designing simple domains to test OPE methods under tabular representation and (ii) we test the same OPE methods and same tabular data generation process, but the input representation for OPE methods is now modified to expose the impact of choosing a different function class for representation. Domains Ultimately, many of the aforementioned factors are intertwined and their usefulness in evaluating OPE performance cannot be considered in isolation. However, they serve as a valuable guide in our selection of benchmark environments. To that end, our benchmark suite includes eight environments. We use two standard RL benchmarks from OpenAI [29]. As many standard RL benchmarks are fixed and deterministic, we design six additional environments that allow control over different design factors. Figure 3.1 depicts one such design factor: the representation complexity. 19 Graph-MC MC !"& !" #"! "! #" Simple % $" Pixel-MC $%& & [position, velocity] !"# !"& # "" !"# ! " !"& Representation Complex Fixed: Stochasticity, Reward Sparcity, Policy Mismatch, Horizon, etc. Figure 3.1: Depicting one of the dimensions which COBS provides control. For the Mountain Car environment, we can select either a tabular, standard coordinate-based, or pixel-based representation of the state while holding other factors fixed. Graph: A flexible discrete environment that can vary in horizon, stochasticity, and sparsity. Graph-POMDP: An extension of Graph to a POMDP setting, where selected information is omitted from the observations that form the behavior data. This enables controlled study of the effect of insufficient representation power relative to other settings in the Graph domain above. Gridworld (GW): A gridworld design that offers larger state and action space than the Graph domains, longer horizon, and similarly flexible design choices for other environmental factors. Using some version of gridworld is standard across many RL experiments. Gridworld enables simple integration of various designs, and fast data collection. Pixel-Gridworld (Pix-GW): A scaled-up domain from Gridworld which enables pixel-based representation of the state space. While such usage is not standard in existing literature, this design offers compelling advantage over many existing standard RL benchmarks. First, this domain enables simple controlled experiments to understand the impact of high-dimensional representation on OPE performance, where the ground truth of various quantities to be estimated is readily obtainable thanks to the access to underlying simpler grid representation. Second, this domain effectively simulates high-dimensional experiments with easily tuned experimental conditions, e.g., degree of stochasticity. This design freedom is not available with many currently standard RL benchmarks. Mountain Car (MC): A standard control domain, which is known to have 20 challenging credit assignment due to sparsity of the reward. Our benchmark for this standard domains allows for function approximation to vary between a linear model and feed-forward neural network, in order to highlight the effects of model misspecification. Pixel Mountain Car (Pix-MC): A modified version of Mountain Car where the state input is pixel-based, testing the methods’ ability to work in high dimensional settings. Tabular Mountain Car (Graph-MC): A simplified version of Mountain Car to a graph, allowing us to complete the test for model misspecification by considering the tabular case. Atari (Enduro): A pixel-based Atari domain. Note that all Atari environments are deterministic and high-dimensional. Instead of choosing many different Atari domains to study, we instead opt to select Enduro as the representative Atari environment, due to its sparsity of reward (and commonly regarded as a highly challenging task). All Atari environments share similar interaction protocol, and can be seamlessly integrated into COBS, if desired. All together, our benchmark consists of 8 environments with characteristics summarized in Table 3.1. Complete descriptions can be found in Appendix A.6. All environments have finite action spaces. Table 3.1: Environment characteristics. Environment Graph Graph-MC MC Pix-MC Enduro G-POMDP GW Pix-GW Is MDP? State desc. T Stoch Env? Stoch Rew? Sparse Rew? Q̂ Class Initial state Absorb. state Frame height Frame skip yes pos. 4 or 16 variable variable variable tabular 0 2T 1 1 yes pos. 250 no no terminal tabular 0 22 1 1 yes pixels 250 no no terminal NN variable img([.5,0]) 2 5 yes pixels 1000 no no dense NN gray img zero img 4 1 no pos. 2 or 8 no no terminal tabular 0 2T 1 1 yes pixels 25 variable no dense NN variable zero img 1 1 yes [pos, vel] 250 no no terminal linear/NN variable [.5,0] 2 5 yes pos. 25 no no dense tabular variable 64 1 1 Experiment Protocol Selection of Policies. We use two classes of policies. The first is stateindependent with some probability of taking any available action. For example, in the Graph environment with two actions, π(a = 0) = p, π(a = 1) = 1 − p where p is a parameter we can control. The second is a state-dependent 21 ϵ−Greedy policy. We train a policy Q∗ (using value iteration or DDQN [85]) and then vary the deviation away from the policy. Hence ϵ − Greedy(Q∗ ) implies we follow a mixed policy π = arg maxa Q∗ (s, a) with probability 1 − ϵ and uniform with probability ϵ. Here ϵ is a parameter we can control. Most OPE methods explicitly require absolute continuity among the policies (πb > 0 whenever πe > 0). Thus, all policies will remain stochastic with this property maintained. Data Generation & Metrics. Each experiment depends on specifying an environment and its properties, behavior policy πb , evaluation policy πe , and number of trajectories N from rolling-out πb for historical data. The true on-policy value V (πe ) is the Monte-Carlo estimate via 10, 000 rollouts of πe . We repeat each experiment m = 10 times with different random seeds. We judge the quality of a method via two metrics: • Relative mean squared error (Relative MSE): m1 Pm 1 V (πe )j )2 Pm (Vb (πe )i − m j=1 P , m 1 2 i=1 (m which allows a fair comparison across different conditions.3 j=1 V (πe )j ) • Near-top Frequency: For each experimental condition, we include the number of times each method is within 10% of the best performing one to facilitate aggregate comparison across domains. Implementation & Hyperparameters. COBS allows running experiments at scale and easy integration with new domains and techniques for future research. The package consists of many domains and reference implementations of OPE methods. Hyperparameters are selected based on publication, code release or author consultation. We maintain a consistent set of hyperparameters for each estimator and each environment across experimental conditions (see hyperparameter choice in appendix Table A.7).4 Baselines OPE methods were historically categorized into importance sampling methods, direct methods, or doubly robust methods. This demarcation was first intro3 metric used in prior OPE work is typically mean squared error: MSE= PThe m 2 b i=1 (V (πe )i − V (πe )i ) . 4 In practice, hyperparameter tuning is not practical for OPE due to a lack of validation signal. 1 m 22 duced for contextual bandits [54], and later extended to the RL setting [97]. Some recent methods have blurred the boundary of these categories. Examples include Retrace(λ) [149] that uses a product of importance weights of multiple time steps for off-policy Q correction, and MAGIC [201] that switches between importance weighting and direct methods. In this benchmark, we propose to group OPE into three similar classes of methods, but with an expanded definition for each category: Inverse Propensity Scoring, Direct Methods, and Hybrid Methods. For the current benchmark, we select representative established baselines from each category. Appendix A.5 contains a full description of all methods under consideration. Inverse Propensity Scoring (IPS) We consider the main four variants: Importance Sampling (IS), Per-Decision Importance Sampling (PDIS), Weighted Importance Sampling (WIS), and Per-Decision WIS (PDWIS). IPS has a rich history in statistics [163, 81, 92], with successful crossover to RL [165]. The key idea is to reweight the rewards in the historical data by the importance sampling ratio between πe and πb , i.e., likelihood of reward under πe versus πb . Direct Methods (DM) While some direct methods make use of importance weight adjustments, a key distinction of direct methods is the focus on regression-based techniques to (more) directly estimate the value functions of the evaluation policy (Qπe or V πe ). This is an area of very active research with rapidly growing literature. We consider 8 different direct approaches, taken from the following respective families of direct estimators. Model-based estimators Perhaps the most commonly used DM is Modelbased (also called approximate model, denoted AM), where the transition dynamics, reward function and termination condition are directly estimated from historical data [97, 161]. The resulting learned MDP is then used to compute the value of πe , e.g., by Monte-Carlo policy evaluation. There are also some recent variants of the model-based estimator, e.g., [228]. Value-based estimators Fitted Q Evaluation (FQE) is a model-free counterpart to AM, and is functionally a policy evaluation counterpart to batch Q learning [123]. Qπ (λ) & Retrace(λ) & Tree-Backup(λ) Several model-free methods originated from off-policy learning settings, but are also natural for OPE. Qπ (λ) [82] can be viewed as a generalization of FQE that looks to the horizon limit to incorporate the long-term value into the backup step. Retrace(λ) [149] and Tree-Backup(λ) [165] also use full trajectories, but additionally incorporate 23 varying levels of clipped importance weights adjustment. The λ-dependent term mitigates instability in the backup step, and is selected based on experimental findings of [149]. Regression-based estimators Direct Q Regression (Q-Reg) & More Robust Doubly-Robust (MRDR) [60] propose two direct methods that make use of cumulative importance weights in deriving the regression estimate for Qπe , solved through a quadratic program. MRDR changes the objective of the regression to that of directly minimizing the variance of the Doubly-Robust estimator. Minimax-style estimators [133] recently proposed a method for the infinite horizon setting — we refer to this estimator as IH. While IH can be viewed as a Rao-Blackwellization of the IS estimator, we include it in the DM category because it solves the Bellman equation for state distributions and requires function approximation, which are more characteristic of DM. IH shifts the focus from importance sampling over action sequences to importance ratio between state density distributions induced by πb and πe . Starting with IH, this style of minimax estimator has recently attracted significant attention in OPE literature, including state-action extension of IH [209, 96] and DICE family of estimators [152, 229, 225, 44]. For our benchmarking purpose, we choose IH as the representative of this family. Hybrid Methods (HM) Hybrid methods subsume doubly robust-like approaches, which combine aspects of both IPS and DM. Standard doubly robust OPE (denoted DR) [97] is an unbiased estimator that leverages DM to decrease the variance of the unbiased estimates produced by importance sampling techniques. Other HM include Weighted Doubly-Robust (WDR) and MAGIC. WDR replaces the importance weights with self-normalized importance weights (similar to WIS). MAGIC introduces adaptive switching between DR and DM; in particular, one can imagine using DR to estimate the value for part of a trajectory and then using DM for the remainder. Using this idea, MAGIC [201] finds an optimal linear combination among a set that varies the switch point b πe (s, a; θ) yields between WDR and DM. Note that any DM that returns Q a set of corresponding DR, WDR, and MAGIC estimators. As a result, we consider 21 hybrid approaches in our experiments. 24 Method MAG. FQE DM FQE IH WDR FQE MAGIC Qπ (λ) Near-top Freq. 30.0% 23.7% 19.0% 17.8% 17.3% Figure 3.2: Left: (Graph domain) Comparing IPS (and IH) under short and long horizon. Mild policy mismatch setting. PDWIS is often best among IPS. But IH outperforms in long horizon. Center: (Pixel-MC) Comparing direct methods in high-dimensional, long horizon setting. Relatively large policy mismatch. FQE and IH tend to outperform. AM is significantly worse in complex domains. Retrace(λ), Q(λ) and Tree-Backup(λ) are very computationally expensive and thus excluded. Right: (Top Methods) The top 5 methods which perform the best across all conditions and domains. 3.2 Empirical Evaluation We evaluate 33 different OPE methods by running thousands of experiments across the 8 domains. Due to limited space, we show only the results from selected environmental conditions in the next section. The full detailed results, with highlighted best method in each class, are available in the appendix. The goal of the evaluation is to demonstrate the flexibility of the benchmark suite to systematically test the different factors of influence. We synthesize the results, and present further considerations and directions for research in Section 3.3. What is the best method? The first important takeaway is that there is no clear-cut winner: no single method or method class is consistently the best performer, as multiple environmental factors can influence the accuracy of each estimator. With that caveat in mind, based on the aggregate top performance metrics, we can recommend from our selected methods the following for each method class (See Figure 3.2 right, appendix Table A.13, and appendix Table A.1). Inverse propensity scoring (IPS). In practice, weighted importance sampling, which is biased, tends to be more accurate and data-efficient than unbiased basic importance sampling methods. Among the four IPS-based estimators, PDWIS tends to perform best (Figure 3.2 left). Direct methods (DM). Generally, FQE, Qπ (λ), and IH tend to perform the best among DM (appendix Table A.1). FQE tends to be more data efficient and is the best method when data is limited (Figure 3.5). Qπ (λ) generalizes 25 Figure 3.3: General Guideline Decision Tree. FQE to multi-step backup, and works particularly well with more data, but is computationally expensive in complex domains. IH is highly competitive in long horizons and with high policy mismatch in a tabular setting (appendix Tables A.5, A.6). In pixel-based domains, however, choosing a good kernel function for IH is not straightforward, and IH can underperform other DM (appendix Table A.9). We provide a comparison among direct methods for tabular (appendix Figure A.13) and complex settings (Figure 3.2 center). Hybrid methods (HM). With the exception of IH, each DM corresponds to three HM: standard doubly robust (DR), weighted doubly robust (WDR), and MAGIC. For each DM, its WDR version often outperforms its DR version. MAGIC can often outperform WDR and DR. However, MAGIC comes with additional hyperparameters, as one needs to specify the set of partial trajectory length to be considered. Unsurprisingly, their performance highly depends on the underlying DM. In our experiments, FQE and Qπ (λ) are typically the most reliable: MAGIC with FQE or MAGIC with Qπ (λ) tend to be among the best hybrid methods (see appendix Figures A.19 - A.23). A recipe for method selection Figure 3.3 summarizes our general guideline for navigating key factors that affect the accuracy of different estimators. To guide the readers through the process, we now dive further into our experimental design to test various factors, and discuss the resulting insights. Do we potentially have representation mismatch? Representation mismatch comes from two sources: model misspecification and poor generalization. Model misspecification refers to the insufficient representation power of the 26 Figure 3.4: Comparing IPS versus Direct methods versus Hybrid methods under short and long horizon, large policy mismatch and large data. Left: (Graph domain) Deterministic environment. Center: (Graph domain) Stochastic environment and rewards. Right: (Graph-POMDP) Model misspecification (POMDP). Minimum error per class is shown. function class used to approximate either the transition dynamics (AM), value function (other DM), or state distribution density ratio (in IH). Having a tabular representation controls for representation mismatch by ensuring adequate function class capacity, as well as zero inherent Bellman error (left branch, Fig 3.3). In such cases, we may suffer from poor generalization without sufficient data coverage, which depends on other domain-specific factors. The effect of representation mismatch (right branch, Fig 3.3) can be understood via two scenarios: • Misspecified and poor generalization: We expose the impact of this severe mismatch scenario via the Graph POMDP construction, where selected information is omitted from an otherwise equivalent Graph MDP. Here, HM substantially outperforms DM (Figure 3.4 right versus left). • Misspecified but good generalization: Function classes such as neural networks have powerful generalization ability, but may introduce bias and inherent Bellman error5 [150, 38] (see linear vs. neural networks comparison for Mountain Car in appendix Fig A.10). Still, powerful function approximation makes (biased) DM very competitive with HM, especially under limited data and in complex domains (see pixel-Gridworld in appendix Fig A.24-A.26). However, function approximation bias may cause serious problems for highdimensional and long horizon settings. In the extreme case of Enduro (very long horizon and sparse rewards), all DM fail to convincingly outperform a naïve average of behavior data (appendix Fig A.9). 5 Inherent Bellman error is defined as supg∈F inf f ∈F ||f − Tπ g||dπ , where F is function class chosen for approximation, and dπ is state distribution induced by evaluation policy π. 27 Figure 3.5: (Gridworld domain) Errors are directly correlated with policy mismatch but inversely correlated with data size. We pick the best direct methods for illustration. The two plots represent the same figure from two different vantage points. Short horizon vs. Long horizon? It is well-known that IPS methods are sensitive to trajectory length [128]. Long horizon leads to an exponential blowup of the importance sampling term, and is exacerbated by significant mismatch between πb and πe . This issue is inevitable for any unbiased estimator [97] (a.k.a., the curse of horizon [133]). Similar to IPS, DM relying on importance weights also suffer in long horizons (appendix Fig A.13), though to a lesser degree. IH aims to bypass the effect of cumulative weighting in long horizons, and indeed performs substantially better than IPS methods in very long horizon domains (Fig 3.2 left). A frequently ignored aspect in previous OPE work is a proper distinction between fixed, finite horizon tasks (IPS focus), infinite horizon tasks (IH focus), and indefinite horizon tasks, where the trajectory length is finite but varies depending on the policy. Many applications should properly belong to the indefinite horizon category.6 Applying HM in this setting requires proper padding of the rewards (without altering the value function in the infinite horizon limit) as DR correction typically assumes fixed length trajectories. How different are behavior and target policies? Similar to IPS, the performance of DM is negatively correlated with the degree of policy mismatch. Figure 3.5 shows the interplay of increasing policy mismatch and historical data (a|s) T size, on the top DM in the deterministic gridworld. We use (supa∈A,x∈X ππeb (a|s) ) as an environment-independent metric of mismatch between the two policies. The performance of the top DM (FQE, Qπ (λ), IH) tend to hold up better than 6 Applying IH in the indefinite horizon case requires setting up an absorbing state that loops over itself with zero terminal reward. 28 IPS methods when the policy gap increases (appendix Figure A.15). FQE and IH are best in the small data regime, and Qπ (λ) performs better as data size increases (Figure 3.5). Increased policy mismatch weakens the DM that use importance weights (Q-Reg, MRDR, Retrace(λ) and Tree-Backup(λ)). Do we have a good estimate of the behavior policy? Often the behavior policy may not be known exactly and requires estimation, which can introduce bias and cause HM to underperform DM, especially in low data regime (e.g., pixel gridworld appendix Figure A.24-A.26). Similar phenomenon was observed in the statistics literature [104]. As the data size increases, HMs regain the advantage as the quality of the πb estimate improves. Is the environment stochastic or deterministic? While stochasticity affects all methods by straining the data requirement, HM are more negatively impacted than DM (Figure 3.4 center, Figure A.14). This can be justified by e.g., the variance analysis of DR, which shows that the variance of the value function with respect to stochastic transitions will be amplified by cumulative importance weights and then contribute to the overall variance of the estimator; see [97, Theorem 1] for further details. We empirically observe that DM frequently outperform their DR versions in the small data case (Figure A.14). In a stochastic environment and tabular setting, HM do not provide significant edge over DM, even in short horizon case. The gap closes as the data size increases (Figure A.14). Challenging common wisdom To illustrate the value of a flexible benchmarking tool, in this section we further synthesize the empirical findings and stress-test several commonly held beliefs about the high-level performance of OPE methods. Is HM always better than DM? No. Overall, DM are surprisingly competitive with HM. Under high-dimensionality, long horizons, estimated behavior policies, or reward/environment stochasticity, HM can underperform simple DM, sometimes significantly (e.g., see appendix Figure A.14). Concretely, HM can perform worse than DM in the following scenarios that we tested: • Tabular with large policy mismatch, or stochastic environments (appendix Figure A.14, Table A.3, A.6). 29 • Complex domains with long horizon and unknown behavior policy (app. Figure A.24-A.26, Table A.8). When data is sufficient, or model misspecification is severe, HM provides consistent gains over DM. Is horizon length the most important factor? No. Despite conventional wisdom suggesting IPS methods are most sensitive to horizon length, we find (a|s) that this is not always the case. Policy divergence supa∈A,x∈X ππeb (a|s) can be just as, if not more, meaningful. For comparison, we designed two scenarios with (a|s) T identical mismatch (supa∈A,x∈X ππeb (a|s) ) as defined in Section 3.2 (see appendix Tables A.11, A.12). Starting from a baseline scenario of short horizon and small policy divergence (appendix Table A.10), extending horizon length leads to 10× degradation in accuracy, while a comparable increase in policy divergence causes a 100× degradation. How good is model-based direct method (AM)? AM can be among the worst performing direct methods (appendix Table A.1). While AM performs well in tabular setting in the large data case (appendix Figure A.13), it tends to perform poorly in high dimensional settings with function approximation (e.g., Figure 3.2 center). Fitting the transition model P (s′ |s, a) is often more prone to small errors than directly approximating Q(s, a). Model fitting errors also compound with long horizons. 3.3 Discussion and Future Directions Finally, we close with a brief discussion on some limitations common to recent OPE benchmarks and more generally OPE experimental studies, and point to areas of development for future studies. Lack of short-horizon benchmark in high-dimensional settings. Evaluation of other complex RL tasks with short horizon is currently beyond the scope of our study, due to the lack of a natural benchmark. For contextual bandits, it has been shown that while DR is highly competitive, it is sometimes substantially outperformed by DM [219]. New benchmark tasks should have longer horizon than contextual bandits, but shorter than typical Atari games. We also currently lack natural stochastic environments in high-dimensional RL benchmarks. An example candidate for medium horizon, complex OPE domain is NLP tasks such as dialogue. 30 Other OPE settings. We outline practically relevant settings that can benefit from benchmarking: • Missing data coverage. A common assumption in the analysis of OPE is a full support assumption: πe (a|s) > 0 implies πb (a|s) > 0, which often ensure unbiasedness of estimators [165, 133, 54]. This assumption is often not verifiable in practice. Practically, violation of this assumption requires regularization of unbiased estimators to avoid ill-conditioning [133, 60]. One avenue to investigate is to optimize the bias-variance trade-off when the full support is not applicable. • Confounding variables. Existing OPE research often assumes that the behavior policy chooses actions solely based on the state. This assumption is often violated when the decisions in the historical data are made by humans instead of algorithms, who may base their decisions on variables not recorded in the data, causing confounding effects. Tackling this challenge, possibly using techniques from causal inference [200, 156], is an important future direction. • Strategic Environmental Behavior. Most OPE methods have focused exclusively on single-agent scenarios under well-defined MDP. Realitic applications of offline RL may have to deal with nonstationary and partial observability induced by strategic behavior from multiple agents [230]. There is currently a lack of a compelling domain to study such a setting. Evaluating new OPE estimators. For our empirical evaluation, we selected a representative set of established baseline approaches from multiple OPE method families. Currently this area of research is very active and as such, new OPE estimators have been and will continue to be proposed. We discuss several new minimax style estimators, notably the DICE family in section 3.1. A minimax-style estimator has also been recently proposed for the model-based regime [214]. Among the ideas that use marginalized state distribution [223] to improve over standard IPS, [102, 103] analyze double reinforcement learning estimator that makes use of both estimates for Q function and state density ratio. While we have not included all estimators in our current benchmark, our software implementation is highly modular and can easily accommodate new estimators and environments. 31 Algorithmic approach to method selection. Using COBS, we showed how to distill a general guideline for selecting OPE methods. However, it is often not easy to judge whether some decision criteria are satisfied (e.g., quantifying model misspecification, degree of stochasticity, or appropriate data size). As more OPE methods continue to be developed, an important missing piece is a systematic technique for model selection, given a relatively high degree of variability among existing techniques. 32 Chapter 4 ADVANCES IN MODEL BASED OPE We present a novel off-policy loss function for learning a transition model in model-based reinforcement learning. Notably, our loss is derived from the off-policy policy evaluation objective with an emphasis on correcting distribution shift. Compared to previous model-based techniques, our approach allows for greater robustness under model mis-specification or distribution shift induced by learning/evaluating policies that are distinct from the data-generating policy. We provide a theoretical analysis and show empirical improvements over existing model-based off-policy evaluation methods. We provide further analysis showing our loss can be used for off-policy optimization (OPO) and demonstrate its integration with more recent improvements in OPO. 4.1 Introduction to Model-Based OPE and OPO We study the problem of learning a transition model in a batch, off-policy reinforcement learning (RL) setting, i.e., of learning a function P (s′ |s, a) from a pre-collected dataset D = {(si , ai , s′i )}ni=1 without further access to the environment. Contemporary approaches to model learning focus primarily on improving the performance of models learned through maximum likelihood estimation (MLE) [193, 48, 113, 43, 42, 137]. The goal of MLE is to pick the model within some model class P that is most consistent with the observed data or, equivalently, most likely to have generated the data. This is done by minimizing negative log-loss (minimizing the KL divergence) summarized as follows: X 1 (4.1) PbMLE = arg min − log(P (s′i |si , ai )). P ∈P n (si ,ai ,s′ )∈D i A key limitation of MLE is that it focuses on picking a good model under the data distribution while ignoring how the model is actually used. In an RL context, a model can be used to either learn a policy (policy learning/optimization) or evaluate some given policy (policy evaluation), without having to collect more data from the true environment. We call this actual 33 objective the “decision problem”. Interacting with the environment to solve the decision problem can be difficult, expensive, and dangerous, whereas a model learned from batch data circumvents these issues. Since MLE (4.1) does not optimize over the distribution of states induced by the policy from the decision problem, it thus does not prioritize solving the decision problem. Notable previous works that incorporate the decision problem into the model learning objective are Value-Aware Model Learning (VAML) and its variants [59, 58, 1]. These methods, however, still define their losses w.r.t. the data distribution as in MLE, and ignore the distribution shift from the pre-collected data to the policy-induced distribution. In contrast, we directly focus on requiring the model to perform well under unknown distributions instead of the data distribution. In other words, we are particularly interested in developing approaches that directly model the batch (offline) learning setting. As such, we ask: “From only pre-collected data, is there a model learning approach that naturally controls the decision problem error?”. We present a new loss function for model learning that: (1) only relies on batch or offline data; (2) takes into account the distribution shift effects; and (3) directly relates to the performance metrics for off-policy evaluation and learning under certain realizability assumptions. The design of our loss is inspired by recent advances in model-free off-policy evaluation [e.g., 134, 208], which we build upon to develop our approach. 4.2 Preliminaries Recall from Section 2.2, in the batch learning setting, we are given a dataset D = {(si , ai , s′i )}ni=1 , where si ∼ dπb (s), ai ∼ πb , and s′i ∼ P (·|si , ai ), where πb is some behavior policy that collects the data. For convenience, we write (s, a, s′ ) ∼ Dπb P , where Dπb (s, a) = dπb (s)πb (a|s). Finally, we need three classes W, V, P of functions. W ⊂ (X → R) represents ratios between state-action occupancies, V ⊂ (S → R) represents value functions and P ⊂ (X → ∆(S)) represents the class of models (or simulators) of the true environment. Note. Any Lemmas or Theorems presented without proof have full proofs in the Appendix. 34 4.3 Minimax Model Learning (MML) for OPE Natural Derivation We start with the off-policy evaluation (OPE) learning objective and derive the MML loss (Def 4.3.1). In Section 4.4, we show the loss also bounds off-policy optimization (OPO) error through its connection with OPE. OPE Decision Problem. Recall from Section 2.2, the OPE objective is to estimate:   ∞ X J(π, P ) ≡ E  γ i ri ∗ i=0 s0 ∼d0 ai ∼π(·|si )  si+1 ∼P ∗ (·|si ,ai ) , ri ∼R(·|si ,ai ) (4.2) the performance of an evaluation policy π in the true environment P ∗ , using only logging data D with samples from Dπb P ∗ . Model-Based OPE. Given a model class P and a desired evaluation policy π, we want to find a simulator Pb ∈ P using only logging data D such that: Pb = arg min |J(π, P ) − J(π, P ∗ )|. P ∈P (4.3) Interpreting Eq. (4.3), we run π in P to compute J(π, P ) as a proxy to J(π, P ∗ ). ∗ If we find some P ∈ P such that |δπP,P | = |J(π, P ) − J(π, P ∗ )| is small, then P is a good simulator for P ∗ . Derivation. Using (2.2) and (2.4), we have: ∗ δπP,P = J(π, P ) − J(π, P ∗ ) = Es∼d0 [VπP (s)] − E(s,a)∼dPπ,γ∗ (·,·) [Er∼R(·|s,a) [r]]. Adding and subtracting E(s,a)∼dPπ,γ∗ [VπP (s)], we have: ∗ δπP,P = Es∼d0 [VπP (s)] − E(s,a)∼dPπ,γ∗ [VπP (s)] + E(s,a)∼dPπ,γ∗ [VπP (s) − Er∼R(·|s,a) [r]]. (4.4) (4.5) To simplify the above expression, we make the following observations. First, Eq. (4.5) can be simplified through the Bellman equation from Eq. (2.3). To ∗ see this, notice that dPπ,γ is equivalent to some d(s)π(a|s) for an appropriate choice of d(s). Thus, E(s,a)∼dPπ,γ∗ [VπP (s) − Er∼R(·|s,a) [r]] = Es∼d(·) [Ea∼π(·|s) [VπP (s) − Er∼R(·|s,a) [r]]] = Es∼d(·) [Ea∼π(·|s) [Es′ ∼P (·|s,a) [γVπP (s)]]] = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )]]. Second, we can manipulate Eq. (4.4) using the definition of dPπ,γ and recursive 35 property of dPπ,t from Eq. (2.5): Es∼d0 [VπP (s)] − E(s,a)∼dPπ,γ∗ [VπP (s)] =− ∞ X γt t=1 ∞ X = −γ Z γ t ∗ dPπ,t (s, a)VπP (s)dν(s, a) Z dPπ,t+1 (s, a)VπP (s)dν(s, a) Z dPπ,t (s̃, ã)P ∗ (s|s̃, ã)π(a|s)VπP (s)dν(s̃, ã, s, a) Z dPπ,t (s, a)P ∗ (s′ |s, a)VπP (s′ )dν(s, a, s′ ) ∗ t=0 = −γ ∞ X γt ∗ t=0 = −γ ∞ X γ t ∗ t=0 = −γE(s,a)∼dPπ,γ∗ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]]. Combining the above allows us to succinctly express: ∗ δπP,P = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )]] − γE(s,a)∼dPπ,γ∗ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]]. ∗ Since D contains samples from Dπb and not dPπ,γ , we use importance sampling ∗ to simplify the right-hand side of δπP,P to: E γ (s,a,s′ )∼D ∗ π P b " P∗ #  dπ,γ  E P P ′ [V (s̃)] − V (s ) . π π Dπb s̃∼P (·|s,a) dP (s,a) (4.6) ∗ Define wπP (s, a) ≡ Dπ,γ . If we knew wπP (s, a) and VπP (for every P ∈ P), πb (s,a) ∗ then we can select a P ∈ P to directly control δπP,P . We encode this intuition as: Definition 4.3.1. [MML Loss] ∀w ∈ W, V ∈ V, P ∈ P, LM M L (w, V, P ) =E(s,a,s′ )∼Dπb (·,·)P ∗ (·|s,a) [w(s, a) ·   Es̃∼P (·|s,a) [V (s̃)] − V (s′ ) ]. When unambiguous, we will drop the MML subscript. ∗ Here we have replaced wπP (s, a) with w coming from function class W and VπP with V from class V. The function class W represents the possible distribution shifts, while V represents the possible value functions. With this intuition, we can formally guarantee that J(π, P ) ≈ J(π, P ∗ ) under the following realizability conditions: 36 VπP VπP (s̃) VπP (s! ) (s̃, ·) ! (s , ·) P P∗ (s, a) ∗ dP π,γ S ×A D πb Figure 4.1: Visual of Eq. (4.6). The error at every point (s, a) in Dπb is the difference between VπP (s̃) (induced by following P ) and VπP (s′ ) (induced by ∗ following P ∗ ). We re-weight the points (s, a) in Dπb to mimic dPπ,γ . Accumulating the errors exactly yields the OPE error of using P as a simulator. MLE, instead, finds a P “pointing” in the same direction as P ∗ for all points in Dπb , ∗ ignoring the discrepancy with dPπ,γ . Assumption 4.1 (Adequate Support). Dπb (s, a) > 0 whenever dPπ,γ (s, a) > 0. dP (s,a) Define wπP (s, a) ≡ Dπ,γ . π (s,a) b Assumption 4.2 (OPE Realizability). For a given π, W × V contains at least ∗ ∗ one of (wπP , VπP ) or (wπP , VπP ) for every P ∈ P. Theorem 4.3.1 (MML & OPE). Under Assumption 4.2, |J(π, Pb ) − J(π, P ∗ )| ≤ γ min max P ∈P w∈W,V ∈V |L(w, V, P )|, (4.7) where Pb = arg minP ∈P maxw∈W,V ∈V |L(w, V, P )|. Remark 4.3.2. We want to choose V, W, P carefully so that many P ∈ P satisfy L(w, V, P ) = 0 and Assumption 4.2. By inspection, L(w, V, P ∗ ) = 0 for any V ∈ V, w ∈ W. Remark 4.3.3. While VπP ∈ V ∀P ∈ P appears strong, it can be verified for every P ∈ P before accessing the data, as the condition does not depend on P ∗ . In principle, we may redesign V to guarantee this condition. Remark 4.3.4. When γ = 0, J does not depend on a transition function, so J(π, P ) = J(π, P ∗ ) ∀P ∈ P. 37 L(w, V, P ∗ ) = 0 and Theorem 4.3.1 implies that the following learning procedure will be robust to any distribution shift in W and any value function in V: Definition 4.3.2 (Minimax Model Learning (MML)). Pb = arg min max P ∈P w∈W,V ∈V |LM M L (w, V, P )|. (4.8) Interpretation and Verifiability Figure 4.1 gives a visual illustration of Eq. (4.6) which leads to the MML Loss (Def 4.3.1). πb has induced an “inbalanced” training dataset Dπb and the importance sampling term acts to rebalance our data because our test dataset ∗ will be dPπ,γ , induced by π. Because the objective is OPE, we don’t mind that P̂ is different than P ∗ so long as EP̂ [VπP̂ ] ≈ EP ∗ [VπP̂ ]. In other words, the size of VπP̂ tells us which state transitions are important to model correctly. We want to appropriately utilize the capacity of our model class P so that P̂ models P ∗ when VπP̂ is large. When it is small, it may be better off to ignore the error in favor of other states. Theorem 4.3.1 quantifies the error incurred by evaluating π in Pb instead of P ∗ , assuming Assumption 4.2 holds. For OPE, Pb is a reasonable proxy for P . In this sense, MML is a principled method approach for model-based OPE. See Appendix B.1 for a complete proof of Thm 4.3.1 and Appendix B.1 for the sample complexity analysis. If the exploratory state distribution dπb and πb are known then Dπb is known. In this case, we can also verify that wπP ∈ W for every P ∈ P a priori. Together with Remark 4.3.3, we may assume that both wπP ∈ W and VπP ∈ V for all ∗ ∗ P ∈ P. Consequently, only one of VπP ∈ V or wπP ∈ W has to be realizable for Theorem 4.3.1 to hold. Instead of checking for realizability apriori, we can perform post-verification that wπPb ∈ W and VπPb ∈ V. Together with the terms depending on P ∗ , realizability of these are also sufficient for Theorem 4.3.1 to hold. This relaxes the strong “for all P ∈ P” condition. Comparison to Model-Free OPE Recent model-free OPE literature [e.g., 134, 208] has similar realizability assumptions to Assumption 4.2. 38 As an example, the method MWL [208] takes the form of: b J(π, P ∗ ) ≈ E(s,a,r)∼Dπb [w(s, a)r] where wb = arg min max |LM W L (w, Q)|, w∈W Q∈Q ∗ requiring QPπ to be realized to be a valid upper bound. Here Q is analogous ∗ ∗ to our function class V where Ea∼π(a|s) [QPπ (s, a)] = VπP (s). The loss LM W L has no dependence on P and is therefore model-free. MQL [208] has analogous realizability conditions to MWL. Our loss, LM M L , has the same realizability assumptions in addition to one related to P (and not P ∗ ). As discussed in Remark 4.3.3, these P-related assumptions can be verified a priori and in principle, satisfied by redesigning the function classes. Therefore, they do not pose a substantial theoretical challenge. See Section 4.6 for a practical discussion. ∗ ∗ An advantage of model-free approaches is that when both wπP , QPπ are realized, they return an exact OPE point estimate. In contrast, MML additionally requires some P ∈ P that makes the loss zero for any w ∈ W, V ∈ V. The advantage of MML is the increased flexibility of a model, enabling OPO (Section 4.4) and visualization of results through simulation (leading to more transparency). While recent model-free OPE and our method both take a minimax approach, the classes W, V, P play different roles. In the model-free case, minimization is w.r.t either W or V and maximization is w.r.t the other. In our case, W, V are on the same (maximization) team, while minimization is over P. This allows us to treat W × V as a single unit, and represents distribution-shifted value ∗ dP functions. A member of this class, Edata [wV ] (= E(s,a)∼Dπb [ Dπ,γ VπP (s)]), ties πb together the OPE estimate. Misspecification of P, V, W Suppose Assumption 4.2 does not hold and P ∗ ̸∈ P. Define a new function h(s, a, s′ ) ∈ H = {w(s, a)V (s′ )|(w, V ) ∈ W × V}. Then we redefine L: L(h, P ) =E(s,a,s′ )∼Dπb (·,·)P ∗ (·|s,a) [ Ex∼P (·|s,a) [h(s, a, x)] − h(s, a, s′ )]. Proposition 4.3.5 (Misspecification discrepancy for OPE). Let H ⊂ (S × A × ∗ S → R) be a set of functions on (s, a, s′ ). Denote (W V )∗ = wπP (s, a)VπP (s′ ) 39 ∗ (or, equivalently, (W V )∗ = wπP (s, a)VπP (s′ )). |J(π, Pb ) − J(π, P ∗ )| ≤ γ min max |L(h, P )| + γϵH , P h∈H (4.9) where ϵH = maxP ∈P minh∈H |L((W V )∗ − h, P )|. L(W V ∗ − h, P ) measures the difference between h and (W V )∗ . Another interpretation of Prop 4.3.5 is if arg maxH∪{(W V )∗ } L(h, P ) = (W V )∗ for some P ∈ P then MML returns a value γϵH below the true upper bound, otherwise the output of MML remains the upperbound. This result illustrates that realizability is sufficient but not necessary for MML to be an upper-bound on the loss. Application to the Online Setting While the main focus of MML is batch OPE and OPO, we will make a few remarks relating to the online setting. In particular, if we assume we can engage in online data collection then W = {1} (the constant function), representing no distribution shift since πb = π. When VAML and MML share the same function class V, we can show that minM maxW,V LM M L (w, V, P )2 ≤ minP LV AM L (V, P ) for any V, M. In other words, MML is a tighter decision-aware loss even in online data collection. In addition, MML enables greater flexibility in the choice of V. See Appendix B.1 for further details. 4.4 Off-Policy Optimization (OPO) Natural Derivation In this section we examine how MML can be integrated into the policy learning/optimization objective. In this setting, the goal is to find a good policy with respect to the true environment P ∗ without interacting with P ∗ . OPO Decision Problem. Given a policy class Π and access to only a logging dataset D with samples from Dπb P ∗ , find a policy π ∈ Π that is competitive with the unknown optimal policy πP∗ ∗ : πb ∗ = arg min |J(π, P ∗ ) − J(πP∗ ∗ , P ∗ )|. π∈Π (4.10) Note: No additional exploration is allowed. Model-Based OPO. Given a model class P, we want to find a simulator Pb ∈ P using only logging data D and subsequently learn πP∗b ∈ Π in Pb through any policy optimization algorithm which we call Planner(·). 40 Algorithm 1 Standard Model-Based OPO Require: D = Dπb P ∗ , Modeler, Planner 1: Learn Pb ← Modeler(D) b P∗ ← Planner(Pb ) 2: Learn π b P∗ 3: return π In Algorithm 1, Modeler(·) refers to any (batch) model learning procedure. The hope for model-based OPO is that the ideal in-simulator policy πP∗b and the actual best (true environment) policy πP∗ ∗ perform competitively: J(πP∗b, P ∗ ) ≈ J(πP∗ ∗ , P ∗ ). Hence, instead of minimizing Eq (4.10) over all π ∈ Π, we can focus Π = {πP∗ }P ∈P . Derivation. Beginning with the objective, we add zero twice: J(πP∗ ∗ , P ∗ ) − J(πP∗ , P ∗ ) = J(πP∗ ∗ , P ∗ ) − J(πP∗ ∗ , P ) {z | } (a) + J(πP∗ ∗ , P ) − J(πP∗ , P ) + J(πP∗ , P ) − J(πP∗ , P ∗ ) . | {z (b) | } {z } (c) ∗ Term (b) is non-positive since πP is optimal in P (πP∗ ∗ is suboptimal), so we can drop it in an upper bound. Term (a) is the OPE estimate of πP∗ ∗ and term (c) the OPE estimate of πP∗ , implying that we should use Theorem 4.3.1. With this intuition, we have: ∗ ∗ Theorem 4.4.1 (MML & OPO). If wπP∗ ∗ , wπPP∗ ∈ W and VπP∗ ∗ , VπPP∗ ∈ V for P P every P ∈ P then: |J(πP∗ ∗ , P ∗ ) − J(πP∗b, P ∗ )| ≤ 2γ min max |L(w, V, P )|. P The statement also holds if, instead, wπP∗ ∗ , wπPP∗ P every P ∈ P. w,V ∗ ∗ ∈ W and VπP∗ ∗ , VπPP∗ ∈ V for P Interpretation and Verifiability Theorem 4.4.1 compares two different policies in the same (true) environment, since πP∗b will be run in P ∗ rather than Pb . In contrast, Theorem 4.3.1 compared the same policy in two different environments. The derivation of Theorem 4.4.1 (see Appendix B.2) shows that having a good bound on the OPE objective is sufficient for OPO. MML shows how to learn a model that exploits this relationship. Furthermore, the realizability assumptions of Theorem 4.4.1 relax the requirements of an OPE oracle. Rather than requiring the OPE estimate for every 41 π, it is sufficient to have the OPE estimate of πP∗ ∗ and πP∗ (for every P ∈ P) when there is a P ∈ P such that L(w, V, P ) is small for any w ∈ W, V ∈ V. We could have instead examined the quantity minπ |J(πP∗ ∗ , P ∗ ) − J(π, P ∗ )| directly from Eq (4.10). What we would find is that the upper bound is 2 minP maxw,V |Ed0 [V ] − L(w, V, P )| and the realizability requirements would ∗ be that VπP ∈ V, wπP ∈ W for every π in some policy class. This is a much stronger requirement than in Theorem 4.4.1. For OPO, apriori verification of realizability is possible by enumerating over P ∈ P. Whereas the target policy π was fixed in OPE, now πP∗ varies for each P ∈ P. It may be more practical to, as in OPE, perform post-verification that wπP∗ ∈ W and VπP∗ ∈ V. If they do not hold, then we can modify the function P b P b classes until they do. This relaxes the “for every P ∈ P” condition and leaves only a few unverifiable quantities relating to P ∗ . Sample complexity and function class misspecification results for OPO can be found in Appendix B.2, B.2. Comparison to Model-Free OPO For minimax model-free OPO, [39] have analyzed a minimax variant of Fitted Q Iteration (FQI) [55], inspired by Antos et al. [12]. FQI is a commonly used model-free OPO method. In addition to realizability assumptions, these methods also maintain a completeness assumption: the function class of interest is closed under bellman update. Increasing the function class size can only help realizability but may break completeness. It is unknown if the completeness assumption of FQI is removable [39]. MML only has realizability requirements. 4.5 Scenarios & Considerations In this section we investigate a few scenarios where we can calculate the class V and W or modify the loss based on structural knowledge of P, W, and V. In examining the scenarios, we aim to verify that MML gives sensible results. For example, in scenarios where we know MLE to be optimal, MML should ideally coincide. Indeed, we show this to be the case for the tabular setting and Linear-Quadratic Regulators. Other scenarios include showing that MML is compatible with incorporating prior knowledge using either a nominal dynamics model or a kernel. The proofs for any Lemmas in this section can be found in Appendix B.4. 42 Linear & Tabular Function Classes When W, V, P are linear function classes then the entire minimax optimization has a closed form solution. In particular, P takes the form P = φ(s, a, s′ )T α where φ ∈ R|S×A×S| is some basis of features with α ∈ R|S×A×S| its parameters and (w(s, a), V (s′ )) ∈ WV = {ψ(s, a, s′ )T β : ∥β∥∞ < +∞} where ψ ∈ R|S×A×S| . Proposition 4.5.1 (Linear Function classes). Let P = φ(s, a, s′ )T α where φ ∈ R|S×A×S| is some basis of features with α its parameters. Let (w(s, a), V (s′ )) ∈ WV = {ψ(s, a, s′ )T β : ∥β∥∞ < +∞}. Then, b = En−T α if En hR Z  φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) En [ψ(s, a, s′ )], (4.11) i φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) has full rank. The tabular setting, when the state-action space is finite, is a common special case. We can choose: ψ(s, a, s′ ) = φ(s, a, s′ ) = ei (4.12) as the ith standard basis vector where i = s|A||S| + a|S| + s′ . There is no model misspecification in the tabular setting (i.e., P ∗ ∈ P), therefore Pb = P ∗ in the case of infinite data. Proposition 4.5.2 (Tabular representation). Let P = φ(s, a, s′ )T α with φ ∈ R|S×A×S| as in Eq (4.12) and α its parameters. Let (w(s, a), V (s′ )) ∈ WV = {φ(s, a, s′ )T β : ∥β∥∞ < +∞}. Assume we have at least one data point from every (s, a) pair. Then: #{(s, a, s′ ) ∈ D} Pbn (s′ |s, a) = . (4.13) #{(s, a, ·) ∈ D} Prop. 4.5.2 shows that MML and MLE coincide, even in the finite-data regime. Both models are simply the observed propensity of entering state s′ from tuple (s, a). Linear Quadratic Regulator (LQR) The Linear Quadratic Regulator (LQR) is defined as linear transition dynamics P ∗ (s′ |s, a) = A∗ s + B ∗ a + w∗ where w∗ is random noise and a quadratic reward function R(s, a) = sT Qs + aT Ra for Q, R ≥ 0 symmetric positive semi-definite. For ease of exposition we assume that w∗ ∼ N (0, σ ∗2 I). We assume that (A∗ , B ∗ ) is controllable. Exploiting the structure of this problem, we can 43 check that every V ∈ V takes the form V (s) = sT U s + q for some symmetric semi-positive definite U and constant q (Appendix Lemma B.4.1). Furthermore, we know controllers of the form π(a|s) = −Ks where K ∈ Rk×n are optimal in LQR [22]. We consider determistic and therefore misspecified models of the form P (s′ |s, a) = As + Ba. W is a Gaussian mixture and we can write LM M L as a function of U, K and (A, B) (Appendix Lemma B.4.2). Proposition 4.5.3 (MML + MLE Coincide for LQR). Let A ∈ Rn×n , B ∈ Rn×k , K ∈ Rk×n . Let U ∈ S n be positive semi-definite. Set k = 1, a single input system. Then, arg min max |LM M L (K, U, (A, B))| = (A∗ , B ∗ ) (A,B) K,U = arg min LM LE (A, B). (A,B) Despite model misspecification, both MLE and MML give the correct paramb B) b = (A∗ , B ∗ ). We leave showing that MML and MLE coincide in eters (A, multi-input (k > 1) LQR systems for future work. Residual Dynamics & Environment Shift Suppose we already had some baseline model P0 of P ∗ . Alternatively, we may view this as the real world starting with (approximately) known dynamics P0 and drifting to P ∗ . We can modify MML to incorporate knowledge of P0 to find the residual dynamics: Definition 4.5.1. [Residual MML Loss] For w ∈ W, V ∈ V, P ∈ P, L(w, V, P ) = E(s,a,s′ )∼Dπb (·,·)P ∗ (·|s,a) [w(s, a) · P0 (x|s, a) − P (x|s, a) Ex∼P0 (·|s,a) [ V (x)] − V (s′ ) ]. P0 (x|s, a) ! This solution form matches the intuition that having prior knowledge in the form of P0 focuses the learning objective on the difference between P ∗ and P0 . Incorporating Kernels Our approach is also compatible with incorporating kernels (which is a way of encoding domain knowledge such as smoothness) to learn in a Reproducing Kernel Hilbert Space (RKHS). For example, we may derive a closed form for max(w,V )∈WV L(w, V, P )2 when W × V corresponds to an RKHS and use standard gradient descent to find Pb ∈ P, making the minimax problem 44 much more tractable. See Appendix B.4 for a detailed discussion on RKHS, computational issues relating to sampling from P and alternative approaches to solving the minimax problem. 4.6 Experiments In our experiments, we seek to answer the following questions: (1) Does MML prefer models that minimize the OPE objective? (2) What can we expect when we have misspecification in V? (3) How does MML perform against MLE and VAML in OPE? (4) Does our approach complement modern offline RL approaches? For this last question, we consider integrating MML with the recently proposed MOREL [106] approach for offline RL. See Appendix B.5 for details on MOREL. Brief Environment Description/Setup We perform our experiments in three different domains. Linear-Quadratic Regulator (LQR). The LQR domain is a 1D environment with stochastic dynamics P ∗ (s′ |s, a). We use a finite class P consisting of deterministic policies. We ensure VπP ∈ V for all P ∈ P by solving the ∗ equations in Appendix Lemma B.4.1. We ensure WπP ∈ W using Appendix Equation (B.7). Cartpole [30]. The reward function is modified to be a function of angle and location rather than 0/1 to make the OPE problem more challenging. Each P ∈ P is a parametrized NN that outputs a mean, and logvariance representing a normal distribution around the next state. We model the class WV as a RKHS as in Prop B.4.3 with an RBF kernel. Inverted Pendulum (IP) [51]. This IP environment has a Runge-Kutta(4) integrator rather than Forward Euler (Runge-Kutta(1)) as in OpenAI [30], producing significantly more realistic data. Each P ∈ P is a deterministic model parametrized with a neural network. We model the class WV as a RKHS as in Prop B.4.3 with an RBF kernel. Further Detail A thorough description of the environments, experimental details, setup, and hyperparameters can be found in Appendix B.5. 45 Figure 4.2: LQR. (Left, OPE Error) MML finds the P ∈ P with the lowest OPE error as P gets richer. Since calculations are done in expectation, no error bars are included. (Right, Verifiability) The OPE error (smoothed) increases with misspecification in V parametrized by ϵ, the expected MSE between the ∗ ∗ true VπP ̸∈ V and the approximated VbπP ∈ V. Nevertheless, directionally they all follow the same trajectory as P gets richer. Results Does MML prefer models that minimize the OPE objective? We vary the size of the model class Figure 4.2 (left) testing to see if MML will pick up on the models which have better OPE performance. When the sizes of |P| are small, each method selects (A∗ , B ∗ ) (e.g. P (s′ |s, a) = A∗ s + B ∗ a), the deterministic version of the optimal model. However, increasing the richness of P, MML begins to pick up on models that are able to better evaluate π. Two remarks are in order. In LQR, policy optimization in (A∗ , B ∗ ) coincides with policy optimization in P ∗ . Therefore, if we tried to do policy optimization in our selected model then our policy would be suboptimal in P ∗ . Secondly, MML favors a model other than (A∗ , B ∗ ) because a good OPE estimate relies on appoximating the contribution from the stochastic part of P ∗ . There is a trade-off between the OPE objective and the OPO objective. MML’s preference is dependent on the capacities of P, W, V. Figure 4.2 (left) illustrates OPE is preferred for W fixed. Appendix Figure B.1 explores the OPO objective and shows that if we increase W then OPO becomes favored. In some sense we are asking MML to be robust to many more OPE problems as |W| ↑ and so the performance on any single one decreases, favoring OPO. What can we expect when we have misspecification in V? To check verifiability in practice, we would run π in a few P ∈ P and calculate VπP . We would check if VπP ∈ V by fitting VbπP and measuring the empirical gap E[(VbπP − VπP )2 ] = ϵ2 . Figure 4.2 (right) shows how MML performs when VπP ̸∈ V but we do have 46 Figure 4.3: (Cartpole, OPE Error) Comparison of model-based approaches for OPE. Lower is better. MML outperforms others. Not pictured: traditional model-free methods such as IS/PDIS have error of order 3-8. VbπP (s) = VπP (s) + N (0, ϵ) ∈ V. Since E[(VbπP − VπP )2 ] = ϵ2 then ϵ is the rootmean squared error between the two functions. Directionally all of the errors go down as |P| ↑, however it is clear that ϵ has a noticeable effect. We speculate that if this error not distributed around zero and instead is dependent on the state then the effects can be worse. How does MML perform against MLE and VAML in OPE? In addition to Figure 4.2 (left), Figure 4.3 also illustrates that our method outperforms the other model-learning approaches in OPE. The environment and reward function is challenging, requiring function approximation. Despite the added complexity of solving a minimax problem, doing so gives nearly an order of magnitude improvement over MLE and many orders over VAML. This validates that MML is a good choice for model-learning for OPE. Algorithm 2 OPO Algorithm (based on MOREL [106]) Require: D, L among {MML, MLE, VAML} 1: Learn an ensemble of dynamics P1 , . . . , P4 ∈ P using Pi = arg minP ∈P L(D) 2: Construct a pessimistic MDP M (see Appendix B.5) with P (s, a) = 1 P4 4 i=1 Pi (s, a). b ← PPO(M) (Best of 3) [180] 3: π Does our approach complement modern offline RL approaches? We integrate MML, VAML, and MLE with MOREL as in Algorithm 2. Consequently, Figure 4.4 shows that MML performs competitively with the other methods, achieving near-optimal performance as the number of trajectories 47 Figure 4.4: (Invert. Pend., OPO Performance) Comparison of model-based approaches for OPO with function-approx using Algorithm 2. Higher is better. MML performs competitively even in low data regimes. increases. MML has good performance even in the low-data regime, whereas other methods perform worse than πb . Performance in the low-data regime is of particular interest since sample efficiency is highly desirable. Algorithm 2 forms a pessimistic MDP where a policy is penalized if it enters a state where there is disagreement between P1 , . . . , P4 . Given that MML performs well in low-data, we can reason that MML produces models with support that stays within the dataset D or generalize well slightly outside this set. The other models poor performance is suggestive of incorrect overconfidence outside of D and PPO produces a policy taking advantage of this. 4.7 Other Related Work Minimax and Model-Based RL. [170] introduce an iterative minimax approach to simultaneously find the optimal-policy and a model of the environment. Despite distribution-shift correction, online data collection is required and is not comparable to MML, where we focus on the batch setting. Batch (Offline) Model-Based RL Recent improvements in batch modelbased RL focus primarily on the issue of policies taking advantage of errors in the model [106, 48, 42, 95]. These improvements typically involve uncertainty quantification to keep the agent in highly certain states to avoid model exploitation. These improvements are independent of the loss function involved. 48 4.8 Discussion and Future Work We have presented a novel approach to learning a model for batch, off-policy model-based reinforcement learning. Our approach follows naturally from the definitions of the OPE and OPO objectives and enjoys distributional robustness and decision-awareness. We examined different scenarios under which our method coincided with other methods as well as when closed form solutions were available. We provided sample complexity analysis and misspecification analysis. Finally, we empirically validated that our method was competitive with current model learning approaches. A key component throughout this chapter has been the function class W × V. Finding other interpretations for this term may prove to be useful outside of MML and is of interest in future work. Furthermore, MML remains part of a two-step OPO pipeline: first learn the model, then return the optimal policy in that model. Another direction of future research is to have a single-shot batch OPO objective that returns both a model and the optimal policy simultaneously, in effect combining MML with the minimax algorithm in [170]. Finally, it may be interesting to integrate MML with other forms of distributionally robust model learning, e.g., Liu et al. [132]. 49 Chapter 5 ADVANCES IN MODEL FREE OPE We propose Fitted Q Evaluation (FQE), a new and simple method for model-free off-policy policy evaluation (OPE) and derive PAC-style bounds. We show experimentally that our OPE method outperforms other popular OPE techniques on a standalone basis, especially in a high-dimensional setting. 5.1 Fitted Q Evaluation (FQE) for Off Policy Evaluation We propose a new and simple model-free technique using function approximation, called Fitted Q Evaluation (FQE). FQE is based on an iterative reductions scheme similar to Fitted Q Iteration (FQI) [56], but for the problem of off-policy evaluation. Algorithm 3 lays out the steps. The key difference with FQI is that the min operator is replaced by Qk−1 (s′i , π(s′i )) (Line 3 of Algorithm 3). Each s′i comes from the original D. Since we know π(s′i ), each e is well-defined. Note that FQE can be augmented with the doubly-robust D k techniques seen in Chapter 3. Algorithm 3 Fitted Q Evaluation: FQE(π, c) Require: Dataset D = {si , ai , s′i , ci }ni=1 ∼ πD . Function class F. Policy π to be evaluated 1: Initialize Q0 ∈ F randomly 2: for k = 1, 2, . . . , K do 3: Compute target yi = ci + γQk−1 (s′i , π(s′i )) ∀i e = {(si , ai ), yi }n 4: Build training set D k i=1 5: Solve a supervised learning problem: P Qk = arg min n1 ni=1 (f (si , ai ) − yi )2 f ∈F 6: return QK (s, π(s)) for any s 5.2 Generalization Guarantee of FQE In this section, we provide sample complexity analysis for FQE. We use the notion of pseudo-dimension as capacity measure of non-linear function class F [65]. Pseudo-dimension dimF , which naturally extends VC dimension into the regression setting, is defined as the VC dimension of the function class induced 50 by the sub-level set of functions of F: dimF = VC-dim({(x, y) 7→ sign(f (x) − y) : f ∈ F}). Pseudo-dimension is finite for a large class of function approximators. For example, Bartlett et al. [18] bounded the pseudo-dimension of piece-wise linear deep neural networks (e.g., with ReLU activations) as O(W L log W ), where W is the number of weights, and L is the number of layers. FQE relies on reductions to supervised learning to update the value function. In off-policy evaluation, the evaluation policy induces a different state-action distribution compared to the data generating distribution µ. We use the notion of concentration coefficient for the worst case, proposed by [147], to measure the degree of distribution shift. The following standard assumption from analysis of related ADP algorithms limits the severity of distribution shift over future time steps: Assumption 5.1 (Concentration coefficient of future state-action distribution). [147, 148, 150, 11, 12, 119, 118, 57, 140] Let P π be the operator acting on f : X → R s.t. (P f )(s, a) = π Z S f (s′ , π(s′ ))P (ds′ |s, a). Given data generating distribution µ, initial state distribution χ, for m ≥ 0 and an arbitrary sequence of stationary policies {πm }m≥1 define the concentration coeffient: d(d0 P π1 P π2 . . . P πm ) βµ (m) = sup . dµ π1 ,...,πm ∞ We assume βµ = (1 − γ)2 P m≥1 mγ m−1 βµ (m) < ∞. This assumption is valid for a fairly large class of MDPs [148]. For instance βµ is finite for any finite MDP, or any infinite state-space MDP with bounded transition density.1 Having a finite concentration coefficient is equivalent the top-Lyapunov exponent Γ ≤ 0 [25], which means the underlying stochastic system is stable. We show below a simple sufficient condition for Assumption 5.1 (albeit stronger than necessary). 1 This assumption ensures sufficient data diversity, even when the executing policy is deterministic. An example of how learning can fail without this assumption is based on the “combination lock” MDP [109]. In this deterministic MDP example, βµ can grow arbitrarily large, and we need an exponential number of samples for both FQE and FQI. See Appendix C.1. 51 Example 5.2.1. Consider an MDP such that for any non-stationary distribution ρ, the marginals over states satisfy µρss(s) ≤ L (i.e., transition dynamics are (s) sufficiently stochastic), and ∃M : ∀s, a : µ(a|s) > M1 (i.e., the behavior policy is sufficiently exploratory). Then βµ ≤ LM . Recall that for a given policy π, the Bellman (evaluation) operator is defined R as (Tπ Q)(s, a) = r(s, a) + γ S Q(s′ , π(s′ ))P (ds′ |s, a). In general Tπ f may not belong to F for f ∈ F. For FQE (and FQI), the main operation in the algorithm is to iteratively project Tπ Qk−1 back to F via Qk = arg minf ∈F ∥f − Tπ Qk−1 ∥. The performance of both FQE and FQI thus depend on how well the function class F approximates the Bellman operator. We measure the ability of function class F to approximate the Bellman evaluation operator via the worst-case Bellman error: Definition 5.2.1 (inherent Bellman evaluation error). Given a function class F and policy π, the inherent Bellman evaluation error of F is defined as dπF = supg∈F inf f ∈F ∥f − Tπ g∥π where ∥·∥π is the ℓ2 norm weighted by the state-action distribution induced by π. We are now ready to state the generalization bound for FQE: Theorem 5.2.1 (Generalization error of FQE). Let the costs be bounded above by C̄. Under Assumption 5.1, for ϵ > 0 &δ ∈ (0, 1), after K iterations of Fitted  4 2 Q Evaluation (Algorithm 3), for n = O Cϵ2 (log Kδ + dimF log Cϵ2 + log dimF ) , we have with probability 1 − δ: V (π) − VK (π) ≤ q γ 1/2 2γ K/2 C
π β (2d + ϵ) + . µ F (1 − γ)3/2 (1 − γ)1/2 e 1 ), compared to O( e 1 ) from other This result shows a dependency on ϵ of O( ϵ2 ϵ4 related ADP algorithms [150, 12]. The price that we pay is a multiplicative constant 2 in front of the inherent error dπF . The error from second term on RHS decays exponentially with iterations K. The proof is in Appendix C.2. 5.3 Empirical Analysis We invite the reader to see Chapter 3 for a full empirical study on the performance of FQE along with many state-of-the-art OPE methods. There will also be additional experiments in Chapter 6 when FQE will be used as a subroutine in policy learning as a means of guaranteeing post-learning performance. Part III POLICY LEARNING WITH GUARANTEES 53 Chapter 6 VALUE-BASED GUARANTEES When learning policies for real-world domains, two important questions arise: (i) how to efficiently use pre-collected off-policy, non-optimal behavior data; and (ii) how to mediate among different competing objectives and constraints. We thus study the problem of batch policy learning under multiple constraints, and offer a systematic solution. We first propose a flexible meta-algorithm that admits any batch reinforcement learning and online learning procedure as subroutines. We then present a specific algorithmic instantiation and provide performance guarantees for the main objective and all constraints. Our algorithm achieves strong empirical results in different domains, including in a challenging problem of simulated car driving subject to multiple constraints such as lane keeping and smooth driving. We also show experimentally that our choice of OPE method (FQE) outperforms other popular OPE techniques on a standalone basis, especially in a high-dimensional setting. 6.1 Introduction We study the problem of policy learning under multiple constraints. Contemporary approaches to learning sequential decision making policies have largely focused on optimizing some cost objective that is easily reducible to a scalar value function. However, in many real-world domains, choosing the right cost function to optimize is often not a straightforward task. Frequently, the agent designer faces multiple competing objectives. For instance, consider the aspirational task of designing autonomous vehicle controllers: one may care about minimizing the travel time while making sure the driving behavior is safe, comfortable, or fuel efficient. Indeed, many such real-world applications require the primary objective function be augmented with an appropriate set of constraints [9]. Contemporary policy learning research has largely focused on either online reinforcement learning (RL) with a focus on exploration, or imitation learning (IL) with a focus on learning from expert demonstrations. However, many real- 54 world settings already contain large amounts of pre-collected data generated by existing policies (e.g., existing driving behavior, power grid control policies, etc.). We thus study the complementary question: can we leverage this abundant source of (non-optimal) behavior data in order to learn sequential decision making policies with provable guarantees on constraint satisfaction? We thus propose and study the problem of batch policy learning under multiple constraints. Historically, batch RL is regarded as a subfield of approximate dynamic programming (ADP) [116], where a set of transitions sampled from the existing system is given and fixed. From an interaction perspective, one can view many online RL methods (e.g., DDPG [129]) as running a growing batch RL subroutine per round of online RL. In that sense, batch policy learning is complementary to any exploration scheme. To the best of our knowledge, the study of constrained policy learning in the batch setting is novel. We present an algorithmic framework for learning sequential decision making policies from off-policy data. We employ multiple learning reductions to online and supervised learning, and present an analysis that relates performance in the reduced procedures to the overall performance with respect to both the primary objective and constraint satisfaction. Constrained optimization is a well studied problem in supervised machine learning and optimization. In fact, our approach is inspired by the work of Agarwal et al. [5] in the context of fair classification. In contrast to supervised learning for classification, batch policy learning for sequential decision making introduces multiple additional challenges. First, setting aside the constraints, batch policy learning itself presents a layer of difficulty, and the analysis is significantly more complicated. Second, verifying whether the constraints are satisfied is no longer as straightforward as passing the training data through the learned classifier. In sequential decision making, certifying constraint satisfaction amounts to an off-policy policy evaluation problem, which is a challenging problem and the subject of active research. In this chapter, we develop a systematic approach to address these challenges, provide a careful error analysis, and experimentally validate our proposed algorithms. In summary, our contributions are: • We formulate the problem of batch policy learning under multiple constraints, and present the first approach of its kind to solve this problem. The definition of constraints is general and can subsume many objectives. Our meta-algorithm utilizes multi-level learning reductions, and we show 55 how to instantiate it using various batch RL and online learning subroutines. We show that guarantees from the subroutines provably lift to provide end-to-end guarantees on the original constrained batch policy learning problem. • Leveraging techniques from batch RL as a subroutine, we provide a refined theoretical analysis for general non-linear function approximation that improves upon the previously known sample complexity bound [150] from O(n4 ) to O(n2 ). • To evaluate and verify constraint satisfaction, we propose a simple new technique for off-policy policy evaluation, which is used as a subroutine in our main algorithm. We show that it is competitive to other off-policy policy evaluation methods. • We validate our algorithm and analysis with two experimental settings. First, a simple navigation domain where we consider safety constraint. Second, we consider a high-dimensional racing car domain with smooth driving and lane keeping constraints. 6.2 Problem Formulation We follow the notation introduced in Section 2.1. Let S ⊂ Rd be a bounded and closed d-dimensional state space. Let A be a finite action space. Let c : X 7→ [0, C] be the primary objective cost function that is bounded by C. Let there be m constraint cost functions, gi : X 7→ [0, G], each bounded by G. To simplify the notation, we view the set of constraints as a vector function g : X 7→ [0, G]m where g(s, a) is the column vector of individual gi (s, a). Let P (·|s, a) denote the (unknown) transition/dynamics model that maps state/action pairs to a distribution over the next state. Let γ ∈ (0, 1) denote the discount factor. Let d0 be the initial states distribution. We consider the discounted infinite horizon setting. An MDP is defined using the tuple (S, A, c, g, P, γ, d0 ). A policy π ∈ Π maps states to actions, i.e., π(s) ∈ A. The value function C π : S 7→ R corresponding to the primary cost P t function c is defined in the usual way: C π (s) = E [ ∞ t=0 γ c(st , at ) | s0 = s], over the randomness of the policy π and transition dynamics P . We similarly define the vector-value function for the constraint costs Gπ : S 7→ Rm as P t Gπ (s) = E [ ∞ t=0 γ g(st , at )|s0 = s]. Define C(π) and G(π) as the expectation of C π (s) and Gπ (s), respectively, over the distribution d0 of initial states. 56 Batch Policy Learning under Constraints In batch policy learning, we have a pre-collected dataset, D = {(si , ai , s′i , c(si , ai ), g1:m (si , ai )}ni=1 , generated from (a set of) historical behavioral policies denoted jointly by πD . The goal of batch policy learning under constraints is to learn a policy π ∈ Π from D that minimizes the primary objective cost while satisfying m different constraints: min C(π) π∈Π (OPT) s.t. G(π) ≤ τ, where G(·) = [g1 (·), . . . , gm (·)]⊤ and τ ∈ Rm is a vector of known constants. We assume that (OPT) is feasible. However, the dataset D might be generated from multiple policies that violate the constraints. Examples of Policy Learning with Constraints Counterfactual & Safe Policy Learning. In conventional online RL, the agent needs to “re-learn” from scratch when the cost function is modified. Our framework enables counterfactual policy learning assuming the ability to compute the new cost objective from the same historical data. A simple example is safe policy learning [70]. Define safety cost G(s, a) = φ(s, a, c) as a new function of existing cost c and features associated with current state-action pair. The goal here is to counterfactually avoid undesirable behaviors observed from historical data. We experimentally study this safety problem in Section 6.5. Other examples from the literature that belong to this safety perspective include planning under chance constraints [158, 23]. The constraint here is G(π) = E[I(s ∈ Serror )] = P(s ∈ Serror ) ≤ τ . Multi-objective Batch Learning. Traditional policy learning (RL or IL) presupposes that the agent optimizes a single cost function. In reality, we may want to satisfy multiple objectives that are not easily reducible to a scalar objective function. One example is learning fast driving policies under multiple behavioral constraints such as smooth driving and lane keeping consistency (see Section 6.5). 57 Equivalence between Constraint Satisfaction and Regularization Our constrained policy learning framework subsumes several existing regularized policy learning settings. Regularization typically encodes prior knowledge, and has been used extensively in the RL and IL literature to improve learning performance. Many instances of regularized policy learning can be naturally cast into (OPT): • Entropy regularized RL [78, 231] is equivalent to G(π) = H(π), where H(π) measures policy entropy. • Smooth imitation learning [122] is equivalent to G(π) = minh∈H ∆(h, π), where H is a class of provably smooth policies and ∆ is a divergence metric. • Regularizing RL with expert demonstration [91] is equivalent to G(π) = E[ℓ(π(x), π ∗ (x))], where π ∗ is the expert policy. • Conservative policy improvement [125, 179, 3] is equivalent to G(π) = DKL (π, πk ), where πk is some “well-behaving” policy. We provide a detailed equivalence derivation of the above examples in Appendix D.1. Of course, some problems are more naturally described using the regularization perspective, while others using constraint satisfaction. More generally, one can establish the equivalence between regularized and constrained policy learning via a simple appeal to Lagrangian duality as shown in Proposition 6.2.1 below. This Lagrangian duality also has a game-theoretic interpretation (Section 5.4 of Boyd and Vandenberghe [26]), which serves as an inspiration for developing our approach. Proposition 6.2.1. Let Π be a convex set of policies. Let C : Π 7→ R, C : Π 7→ RK be value functions. Consider the two policy optimization tasks: Regularization: min C(π) + λ⊤ G(π) Constraint: min C(π) π∈Π π∈Π s.t. G(π) ≤ τ. Assume that the Slater’s condition is satisfied in the Constraint problem (i.e., ∃π s.t. G(π) < τ ). Assume also that the constraint cannot be removed without changing the optimal solution, i.e., inf π∈Π C(π) < inf π∈Π:G(π)≤τ C(π). Then ∀ λ > 0, ∃ τ , and vice versa, such that Regularization and Constraint share the same optimal solutions. (Proof in Appendix D.1.) 58 Algorithm 4 Meta-algo for Batch Constrained Learning 1: for each round t do 2: πt ← Best-response(λt ) P b t ← 1 Pt′ λ ′ bt ← 1t tt′ =1 πt′ , λ 3: π t =1 t t bt , λ) 4: Lmax = maxλ L(π b t ), λ bt) 5: Lmin = L(Best-response(λ 6: if Lmax − Lmin ≤ ω then bt 7: Return π 8: λt+1 ← Online-algorithm(π1 , . . . , πt−1 , πt ) 6.3 Proposed Approach To make use of strong duality, we first convexify the policy class Π by allowing stochastic combinations of policies, which effectively expands Π into its convex hull Conv(Π). Formally, Conv(Π) contains randomized policies,1 which we P P denote π = Tt=1 αt πt for πt ∈ Π and Tt=1 αt = 1. Executing a mixed π consists of first sampling one policy πt from π1:T according to distribution α1:T , P and then executing πt . Note that we still have E[π] = Tt=1 αt E[πt ] for any first-moment statistic of interest (e.g., state distribution, expected cost). It is easy to see that the augmented version of (OPT) over Conv(Π) has a solution at least as good as the original (OPT). As such, to lighten the notation, we will equate Π with its convex hull for the rest of the chapter. Meta-Algorithm The Lagrangian of (OPT) is L(π, λ) = C(π) + λ⊤ (G(π) − τ ) for λ ∈ Rm +. Clearly (OPT) is equivalent to the min-max problem: min maxk L(π, λ). We π∈Π λ∈R + assume (OPT) is feasible and that Slater’s condition holds (otherwise, we can simply increase the constraint τ by a tiny amount). Slater’s condition and policy class convexification ensure that strong duality holds [26], and (OPT) is also equivalent to the max-min problem: maxk min L(π, λ). λ∈R+ π∈Π Since L(π, λ) is linear in both λ and π, strong duality is also a consequence of von Neumann’s celebrated convex-concave minimax theorem for zero-sum games [217]. From a game-thoeretic perspective, the problem becomes finding the equilibrium of a two-player game between the π−player and the λ−player [64]. In this repeated game, the π−player minimizes L(π, λ) given the current λ, and the λ−player maximizes it given the current (mixture over) π. We first present a meta-algorithm (Algorithm 4) that uses any no-regret online 1 This places no restrictions on the individual policies. Individual policies can be arbitrarily non-convex. Convexifiying a policy class amounts to allowing ensembles of learned policies. 59 learning algorithm (for λ) and batch policy optimization (for π). At each iteration, given λt , the π-player runs Best-response to get the best response: Best-response(λt ) = arg min L(π, λt ) π∈Π = arg min C(π) + λ⊤ t (G(π) − τ ). π∈Π This is equivalent to a standard batch reinforcement learning problem where we learn a policy that is optimal with respect to c + λ⊤ t g. The corresponding mixed strategy is the uniform distribution over all previous πt . In response to the π−player, the λ−player employs Online-algorithm, which can be any no-regret algorithm that satisfies: X t L(πt , λt ) ≥ max λ X L(πt , λ) − o(T ). t Finally, the algorithm terminates when the estimated primal-dual gap is below a threshold ω (Lines 7-8). Leaving aside (for the moment) issues of generalization, Algorithm 4 is guaranteed to converge assuming: (i) Best-response gives the best single policy in the class, and (ii) Lmax and Lmin can be evaluated exactly. Proposition 6.3.1. Assuming (i) and (ii) above, Algorithm 4 is guaranteed to stop and the convergence depends on the regret of Online-algorithm. (Proof in Appendix D.2.) Our Main Algorithm We now focus on a specific instantiation of Algorithm 4. Algorithm 5 is our main algorithm in this chapter. Policy Learning. We instantiate Best-response with Fitted Q Iteration (FQI), a model-free off-policy learning approach [56]. FQI relies on a series of reductions to supervised learning. The key idea is to approximate the true action-value function Q∗ by a sequence {Qk ∈ F}K k=0 , where F is a chosen function class. In Lines 3 & 9, FQI(c+λ⊤ g) is defined as follows. With Q0 randomly initialized, e = {(s , a ), y }n for each k = 1, . . . , K, we form a new training dataset D k i i i i=1 where: ∀i : yi = (ci + λ⊤ gi ) + γ min Qk−1 (s′i , a), a 60 Algorithm 5 Constrained Batch Policy Learning Require: Dataset D = {si , ai , s′i , ci , gi }ni=1 ∼ πD . Online algorithm parameters: ℓ1 norm bound B, learning rate η B B 1: Initialize λ1 = ( m+1 , . . . , m+1 ) ∈ Rm+1 2: for each round t do 3: Learn πt ← FQI(c + λ⊤ // FQI with cost c + λ⊤ t g) t g b 4: Evaluate C(πt ) ← FQE(πt , c) // Algo 3 with πt , cost c b t ) ← FQE(πt , g) 5: Evaluate G(π // Algo 3 with πt , cost g P bt ← 1t tt′ =1 πt′ 6: π P P b π b ′ ), G( b π b ′) bt ) ← 1t tt′ =1 G(π bt ) ← 1t tt′ =1 C(π 7: C( t t P b t ← 1 t′ λ ′ 8: λ t t =1 t b ⊤ g) b⊤g e ← FQI(c + λ 9: Learn π // FQI with cost c + λ t t b b e) ← e e e 10: Evaluate C(π FQE( π , c), G( π ) ← FQE( π , g)   h b π b π bt ) + λ⊤ (G( bt ) − τ )⊤ , 0 C( 11: b max = L 12: 1 h i⊤ ⊤, 0 b ⊤ (G( b b b π e) + λ e Lmin = C(π ) − τ ) t b b if Lmax − Lmin ≤ ω then 13: 14: max λ,∥λ∥ =B bt Returnh π i⊤ 15: b t ) − τ )⊤ , 0 Set zt = (G(π 16: λt+1 [i] = B Pλtλ[i]e[j]e−ηzt [j] ∀i −ηzt [i] j t i⊤ ∈ Rm+1 // λ[i] the ith coordinate and (si , ai , s′i , ci , gi ) ∼ D (original dataset). A supervised regression procedure is called to solve for: n 1X Qk = arg min (f (si , ai ) − yi )2 . n i=1 f ∈F The policy then: πK = arg mina QK (·, a). FQI has been shown to work well with several empirical domains: spoken dialogue systems [162], physical robotic soccer [173], and cart-pole swing-up [172]. Another possible model-free subroutine is Least-Squares Policy Iteration (LSPI) [115]. One can also consider model-based alternatives [159]. Off-policy Policy Evaluation. A crucial difference between constrained policy learning and existing work on constrained supervised learning is the technical challenge of evaluating the objective and constraints. First, estimating b b b L(π, λ) (Lines 11-12) requires estimating C(π) and G(π). Second, any gradientb based approach to Online-learning requires passing in G(π) − τ as part of gradient estimate (line 15). This problem is known as the off-policy policy b b evaluation (OPE) problem: we need to evaluate C(π) and G(π) having only access to data D ∼ πD . We studied existing OPE techniques in Chapter 3, and proposed two methods in subsequent chapters. We will be using FQE, from 61 Chapter 5, to instantiate Algorithm 5 Online Learning Subroutine. As L(πt , λ) is linear in λ, many online convex optimization approaches can be used for Online-algorithm. Perhaps the simpliest choice is Online Gradient Descent (OGD) [234]. We include an instantiation using OGD in Appendix D.6. For our main Algorithm 5, similar to [5],qwe use Exponentiated Gradient (EG) √ [108], which has a regret bound of O( log(m)T ) instead of O( mT ) as in OGD. One can view EG as a variant of Online Mirror Descent [153] with a softmax link function, or of Follow-the-Regularized-Leader with entropy regularization [182]. Gradient-based algorithms generally require bounded λ. We thus force ∥λ∥1 ≤ B using hyperparameter B. Solving (OPT) exactly requires B = ∞. We will analyze Algorithm 5 with respect to finite B. With some abuse of notation, we augment λ into a (m + 1)−dimensional vector by appending B − ∥λ∥1 , and augment the constraint cost vector g by appending 0 (Lines 11, 12 & 15 of Algorithm 5).2 6.4 Theoretical Analysis Convergence Guarantee The convergence rate of Algorithm 5 depends on the radius B of the dual variables λ, the maximal constraint value G, and the number of constraints m. 2 In particular, we can show O( Bω2 ) convergence for primal-dual gap ω. Theorem 6.4.1 (Convergence of Algorithm 5). After T iterations, the empirical duality gap is bounded by B log(m + 1) b b L + 2ηBG2 . max − Lmin ≤ 2 ηT Consequently, to achieve the primal-dual gap of ω, setting η = 4Gω2 B will ensure 2 2 that Algorithm 5 converges after 16B G ωlog(m+1) iterations. (Proof in Appendix 2 D.2.) Convergence analysis of our main Algorithm 5 is an extension of the proof to Proposition 6.3.1, leveraging the no-regret property of the EG procedure [182]. The (m + 1)th coordinate of g is thus always satisfied. This augmentation is only necessary when executing EG. 2 62 Generalization Guarantee of FQI We recall a few definitions from Chapter 5, including that for concentration of the state-action distribution and inherent bellman error. Assumption 6.1 (Concentration coefficient of future state-action distribution). [147, 148, 150, 11, 12, 119, 118, 57, 140] Let P π be the operator acting on f : X 7→ R s.t. (P f )(s, a) = π Z S f (x′ , π(x′ ))P (ds′ |s, a). Given data generating distribution µ, initial state distribution d0 , for m ≥ 0 and an arbitrary sequence of stationary policies {πm }m≥1 define the concentration coefficient: d(d0 P π1 P π2 . . . P πm ) βµ (m) = sup . dµ π1 ,...,πm ∞ We assume βµ = (1 − γ)2 P m≥1 mγ m−1 βµ (m) < ∞. Definition 6.4.1 (Inherent Bellman evaluation error). Given a function class F and policy π, the inherent Bellman evaluation error of F is defined as dπF = supg∈F inf f ∈F ∥f − Tπ g∥π where ∥·∥π is the ℓ2 norm weighted by the state-action distribution induced by π. We can show, analogous to FQE, a generalization bound for FQI. While FQI has been widely used, to the best of our knowledge, a complete analysis of FQI for non-linear function approximation has not been previously reported.3 Definition 6.4.2 (Inherent Bellman optimality error). [150] Recall that the Bellman optimality operator is defined as: (TQ)(s, a) = r(s, a) + γ Z min Q(s′ , a′ )P (ds′ |s, a). S a′ ∈A Given a function class F, the inherent Bellman error is defined as dF = supg∈F inf f ∈F ∥f − Tg∥µ , where ∥·∥µ is the ℓ2 norm weighted by µ, the stateaction distribution induced by πD . Theorem 6.4.2 (Generalization error of FQI). Under Assumption 6.1, for ϵ > 0  4 & δ ∈ (0, 1), after K iterations of Fitted Q Iteration, for n = O Cϵ2 (log Kδ + 3 FQI for continuous action space from [11] is a variant of fitted policy iteration and not the version of FQI under consideration. The appendix of [120] contains a proof of FQI but for linear functions. 63 2  dimF log Cϵ2 + log dimF ) , we have with probability 1 − δ:  2γ q K/2 β (2d + ϵ) + 2γ C , C ∗ − C(πK ) ≤ µ F (1 − γ)3 where πK is the policy acting greedy with respect to the returned function QK . (Proof in Appendix D.5.) End-to-End Generalization Guarantee We are ultimately interested in the test-time performance and constraint satisfaction of the returned policy from Algorithm 5. We now connect the previous analyses from Theorems 6.4.1, 5.2.1 & 6.4.2 into an end-to-end error analysis. Since Algorithm 5 uses FQI and FQE as subroutines, the inherent Bellman error terms dF and dπF will enter our overall performance bound. Estimating the inherent Bellman error caused by function approximation is not possible in general (chapter 11 of Sutton and Barto [191]). Fortunately, a sufficiently expressive F can generally make dF and dπF to arbitrarily small. To simplify our end-to-end analysis, we assume dF = 0 and dπF = 0, i.e., the function class F is closed under applying the Bellman operator.4 Assumption 6.2. We consider function classes F sufficiently rich so that ∀f : Tf ∈ F & Tπ f ∈ F for the policies π returned by Algorithm 5. With Assumptions 6.1 & 6.2, we have the following error bound: Theorem 6.4.3 (Generalization guarantee of Algorithm 5). Let π ∗ be the optimal policy to (OPT). Denote V = C + BG. Let K be the number of iterations of FQE and FQI. Let πb be the policy returned by Algorithm 5, with  4 termination threshold ω. For ϵ > 0 & δ ∈ (0, 1), when n = O Vϵ2 (log K(m+1) + δ 2  dimF log Vϵ2 + log dimF ) , we have with probability at least 1 − δ: C(πb ) ≤ C(π ∗ ) + ω + and G(πb ) ≤ τ + 2  (4 + B)γ q K/2 β ϵ + 2γ V , µ (1 − γ)3 V +ω γ 1/2 q 2γ K/2 V  + β ϵ + . µ B (1 − γ)3/2 (1 − γ)1/2 The proof is in Appendix D.3. This result guarantees that, upon termination of Algorithm 5, the true performance on the main objective can be arbitrarily 4 A similar assumption was made in Cheng et al. [40] on near-realizability of learning the model dynamics. 64 close to that of the optimal policy. At the same time, each constraint will be approximately satisfied with high probability, assuming sufficiently large B & K, and sufficiently small ϵ. 6.5 Empirical Analysis We perform experiments on two different domains: a grid-world domain (from OpenAI’s FrozenLake) under a safety constraint, and a challenging highdimensional car racing domain (from OpenAI’s CarRacing) under multiple behavior constraints. We seek to answer the following questions in our experiments: (i) whether the empirical convergence behavior of Algorithm 5 is consistent with our theory, and (ii) how the returned policy performs with respect to the main objective and constraint satisfaction. Appendix D.7 includes a more detailed discussion of our experiments. Frozen Lake. Environment & Data Collection. The environment is an 8x8 grid. The agent has 4 actions N,S,E,W at each state. The main goal is to navigate from a starting position to the goal. Each episode terminates when the agent reaches the goal or falls into a hole. The main cost function is defined as c = −1 if goal is reached, otherwise c = 0 everywhere. We simulate a non-optimal data gathering policy πD by adding random sub-optimal actions to the shortest path policy from any given state to goal. We run πD for 5000 trajectories to collect the behavior dataset D (with constraint cost measurement specified below). Counterfactual Safety Constraint. We augment the main objective c with safety constraint cost defined as g = 1 if the agent steps into a hole, and g = 0 otherwise. We set the constraint threshold τ = 0.1, roughly 75% of the accumulated constraint cost of behavior policy πD . The threshold can be interpreted as a counterfactually acceptable probability that we allow the learned policy to fail. b b Results. The empirical primal dual gap L max − Lmin in Figure 6.1 (left) quickly decreases toward the optimal gap of zero. The convergence is fast and monotonic, supporting the predicted behavior from our theory. The testtime performance in Figure 6.1 (middle) shows the safety constraint is always satisfied, while the main objective cost also smoothly converges to the optimal value achieved by an online RL baseline (DQN) trained without regard to the constraint. The returned policy significantly outperformed the data gathering 65 Figure 6.1: FrozenLake Results. (Left) Empirical duality gap of algorithm 5 vs. optimal gap. (Middle) Comparison of returned policy and others w.r.t. (top) main objective value and (bottom) safety constraint value. (Right) FQE vs. other OPE methods on a standalone basis. Figure 6.2: CarRacing Results. (Left) & (Middle) (Lower is better) Comparing our algorithm, regularized LSPI, online RL w/o constraints, behavior policy πD w.r.t. main cost objectives and two constraints. (Right) FQE vs. other OPE methods on a standalone basis. policy πD on both main and safety cost. Car Racing. Environment & Data Collection. The car racing environment, seen in Figure D.1 (right), is a high-dimensional domain where the state is a 96 × 96 × 3 tensor of raw pixels. The action space A = {steering × gas × brake} takes 12 discretized values. The goal in this episodic task is to traverse over 95% of the track, measured by a given number of “tiles” as a proxy for distance coverage. The agent receives a reward (negative cost) for each unique tile crossed and no reward if the agent is off-track. A small positive cost applies at every time step, with maximum horizon of 1000 for each episode. With these costs given by the environment, one can train online RL agent using DDQN [211]. We collect ≈ 1500 trajectories from DDQN’s randomization, resulting in data set D with ≈ 94000 transition tuples. Fast Driving under Behavioral Constraints. We study the problem of minimizing environment cost while subject to two behavioral constraints: smooth driving and lane centering. The first constraint G0 approximates 66 smooth driving by g0 (s, a) = 1 if a contains braking action, and 0 otherwise. The second constraint cost g1 measures the distance between the agent and center of lane at each time step. This is a highly challenging setup since three objectives and constraints are in direct conflict with one another, e.g., fast driving encourages the agent to cut corners and apply frequent brakes to make turns. Outside of this work, we are not aware of previous work in policy learning with 2 or more constraints in high-dimensional settings. Baseline and Procedure. As a naïve baseline, DDQN achieves low cost, but exhibits “non-smooth” driving behavior (see our supplementary videos). We set the threshold for each constraint to 75% of the DDQN benchmark. We also compare against regularized batch RL algorithms [57], specifically regularized LSPI. We instantiate our subroutines, FQE and FQI, with multilayered CNNs. We augment LSPI’s linear policy with non-linear features derived from a well-performing FQI model. Results. The returned mixture policy from our algorithm achieves low main objective cost, comparable with online RL policy trained without regard to constraints. After several initial iterations violating the braking constraint, the returned policy — corresponding to the appropriate λ trade-off — satisties both constraints, while improving the main objective. The improvement over data gathering policy is significant for both constraints and main objective. Regularized policy learning is an alternative approach to (OPT) (section 6.2). We provide the regularized LSPI baseline the same set of λ found by our algorithm for one-shot regularized learning (Figures 6.2 (left & middle)). While regularized LSPI obtains good performance for the main objective, it does not achieve acceptable constraint satisfaction. By default, regularized policy learning requires parameter tuning heuristics. In principle, one can perform a grid-search over a range of parameters to find the right combination — we include such an example for both regularized LSPI and FQI in Appendix D.7. However, since our objective and constraints are in conflict, main objective and constraint satisfaction of policies returned by one-shot regularized learning are sensitive to step changes in λ. In constrast, our approach is systematic, and is able to avoid the curse-of-dimensionality of brute-force search that comes with multiple constraints. In practice, one may wish to deterministically extract a single policy from the returned mixture for execution. A de-randomized policy can be obtained 67 naturally from our algorithm by selecting the best policy from the existing FQE’s estimates of individual Best-response policies. Off-Policy Evaluation The off-policy evaluation by FQE is critical for updating policies in our algorithm, and is ultimately responsible for certifying constraint satisfaction. While other OPE methods can also be used in place of FQE, we find that the estimates from popular methods are not sufficiently accurate in a highdimensional setting. Complementing the more extensive ablations study in Chapter 3, we select an individual policy and compare FQE against PDIS [164], DR [98] and WDR [202] with respect to the constraint cost evaluation. To compare both accuracy and data-efficiency, for each domain we randomly sample different subsets of dataset D (from 10% to 100% transitions, 30 trials each). Figure 6.1 (right) and 6.2 (right) illustrate the difference in quality. In the FrozenLake domain, FQE performs competitively with the top baseline method (DR and WDR), converging to the true value estimate as the data subsample grows close to 100%. In the high-dimensional car domain, FQE signficantly outperforms other methods. 6.6 Other Related Work Constrained MDP (CMDP). The CMDP is a well-studied problem [9]. Among the most important techniques for solving CMDP are the Lagrangian approach and solving the dual LP program via occupation measure. However, these approaches only work when the MDP is completely specified, and the state dimension is small such that solving via an LP is tractable. More recently, the constrained policy optimization approach (CPO) by [3] learns a policy when the model is not initially known. The focus of CPO is on online safe exploration, and thus is not directly comparable to our setting. Multi-objective Reinforcement Learning. Another related area is multiobjective reinforcement learning (MORL)[212, 174]. Generally, research in MORL has largely focused on approximating the Pareto frontier that trades-off competing objectives [212, 174]. The underlying approach to MORL frequently relies on linear or non-linear scalarization of rewards to heuristically turns the problem into a standard RL problem. Our proposed approach represents another systematic paradigm to solve MORL, whether in batch or online settings. 68 6.7 Discussion We have presented a systematic approach for batch policy learning under multiple constraints. Our problem formulation can accommodate general definition of constraints, as partly illustrated by our experiments. We provide guarantees for our algorithm for both the main objective and constraint satisfaction. Our strong empirical results show a promise of making constrained batch policy learning applicable for real-world domains, where behavior data is abundant. Our implementation complies with the steps laid out in Algorithm 5. In very large scale or high-dimensional problems, one could consider a noisy update version for both policy learning and evaluation. We leave the theorerical and practical exploration of this extension to future work. 69 Chapter 7 LTL-BASED GUARANTEES IN DISCRETE DOMAINS We study the problem of policy optimization (PO) with linear temporal logic (LTL) constraints. The language of LTL allows flexible description of tasks that may be unnatural to encode as a scalar cost function. We consider LTL-constrained PO as a systematic framework, decoupling task specification from policy selection, and as an alternative to the standard of cost shaping. With access to a generative model, we develop a modelbased approach that enjoys a sample complexity analysis for guaranteeing both task satisfaction and cost optimality (through a reduction to a reachability problem). Empirically, our algorithm can achieve strong performance even in low-sample regimes. In this chapter, we propose a novel policy optimization framework for RL under LTL constraints. Our approach relies on two assumptions that are significantly less restrictive than those in prior work and circumvent the negative results on RL-modulo-LTL: the availability of a generative model of the environment and a lower bound on the transition probabilities in the underlying MDP. Under these assumptions, we derive a learning algorithm based on a reduction to a reachability problem. The reduction in our method can be instantiated with several planning procedures that handle unknown dynamics [21, 166]. We show that our algorithm offers strong constraint satisfaction guarantees and give a rigorous sample complexity analysis of the algorithm. In summary, the contributions are: 1) We provide a novel approach to LTL-constrained RL that requires significantly fewer assumptions, and offers stronger guarantees, than previous work. 2) We develop several new theoretical tools for our analysis. These may be of independent interest. 3) We empirically validate using both infinite- and indefinite-horizon problems, and with composite specifications such as collecting items while avoiding 70 enemies. We find that our method enjoys strong performance, often requiring many fewer samples than our worst-case guarantees. 7.1 Motivating Examples R 1 2 p 1-p S Figure 7.1: (Left) Infinite Loop. The robot must perpetually walk between the coffee room and office. Without proper state-space augmentation, a markovian cost function cannot capture this task. (Right) Safe Delivery. A packet must be delivered without being interfered. Policy 2 should be chosen. One would need to penalize receiving the packet significantly over having it stolen: R > S. We examine two examples where standard cost engineering cannot capture the task (Figure 7.1). We consider the undiscounted setting here. See [131, 2] for difficult examples for the discounted setting. Example 1 (Infinite Loop). A robot is given the task of perpetually walking between the coffee room and the office (Figure 7.1 (Left)). To achieve this behavior, both the policy and cost-function must be history-dependent. These can be made Markovian through proper state-space augmentation and has been studied in hierarchical reinforcement learning or learning with options [121, 189]. Options engineering is laborious and requires expertise. Nevertheless, without the appropriate augmentation, any cost-optimal policy of a Markovian cost function will fail at the task. We will see in Section 7.2 that any LTL expression comes with automatic state-space augmentation, requiring no expert input. Example 2 (Safe Delivery). The goal is to maximize the probability of safely sending a packet from one computer to another (Figure 7.1 (Right)). Policy 1 leads to a hacker sniffing packets but passing them through, and is unsafe. Policy 2 leads to a hacker stealing packets with probability p > 0, and is safe with probability 1 − p, and is the policy that satisfies the task. For cost engineering, let R and S be the recurring costs of the received and stolen states. For the two policies, the avg. costs are g1 = R and g2 = pS + (1 − p)R. Strangely, we must set R > S in order for g2 < g1 . Fortunately, optimizing any cost function constrained to satisfying the LTL specification does not suffer 71 from this counter intuitive behavior as only policy 2 has any chance of satisfying the LTL expression. 7.2 Background and Problem Formulation For an introduction to the background for LTL, see Section 2.3. From there, recall a few key definitions. We construct a Product MDP from syncronizing a Labelled MDP M with a specification-specific automaton (LDBA) B: Definition 7.2.1. (Product MDP) XM,B = (S, A, P, C, d0 , L, S ∗ ) is an MDP with S = S M × S B , A = AM ∪ AB , C((m, b), a) = C M (m, a) if a ∈ AM (m) B B M M otherwise 0, d0 = {(m, b)|m ∈ dM 0 , b ∈ P (s0 , L (m))}, L((m, b)) = L (m), S ∗ = {(·, b) ∈ S|b ∈ S B∗ } accepting states, and P : S × A → ∆(S) taking the form:    P M (m, a, m′ )    a ∈ AM (m), b′ ∈ P B (b, L(m′ ))    0, otherwise P ((m, b), a, (m′ , b′ )) = 1, a ∈ AB (b), b′ ∈ P B (b, a), m = m′ . A run τ = (s0 , s1 , . . .) = ((m0 , b0 ), (m1 , b1 ), . . .) in X is accepting (accepted) if (b0 , b1 , . . .), the projection onto B, is accepted. Equivalently, some s ∈ S ∗ in X is visited infinitely often. This leads us to the following definition of LTL satisfaction: Definition 7.2.2 (Satisfaction, τ |= φ). A run τ in X satisfies φ, denoted τ |= φ, if it is accepted. Definition 7.2.3. (Satisfaction, π |= φ) A policy satisfies φ with probability P [π |= φ] = Eτ ∼TPπ [1τ |=φ ]. Here, 1X is an indicator variable which is 1 when X is true, otherwise 0. TPπ is the set of trajectories induced by π in X with transition function P . Problem Formulation Our goal is to find a policy that simultaneously satisfies a given LTL specification φ with highest probability (probability-optimal) and is also optimal w.r.t. the cost function of the MDP. We consider (stochastic) Markovian policies Π, and define the set of all probability-optimal policies as Πmax = {arg maxπ′ ∈Π P [π ′ |= φ]}. We first define the gain g (average-cost) and transient 72 cost J: −1 1 TX lim C(st , π(st )) τ |= φ , T →∞ T t=0 " gπP ≡Eτ ∼TPπ JπP ≡Eτ ∼TPπ # "κ τ X # (7.1) C(st , π(st )) τ |= φ , t=0 where κτ is the first (hitting) time the trajectory τ leaves the transient states induced by π. When P is clear from context, we abbreviate gπP and JπP by gπ and Jπ , respectively. Gain optimality for infinite horizon problems has a long history in RL [21, 166]. Complementary to gain optimality, we consider a hybrid objective including the transient cost. For any λ ≥ 0, we define the optimal policy as the probabilityoptimal policy with minimum combined cost: πλ∗ ≡ arg min Jπ + λgπ (LTL-OPT) π∈Πmax = arg min (Jπ + λgπ )P [π |= φ] π∈Πmax P (≡ Vπ,λ ). In other words, probability-optimal policies are those that satisfy the entirety of P the task, both desired and required behaviors, where Vπ,λ ≡ (Jπ +λgπ )P [π |= φ] is the normalized value function1 , corresponding to a notion of energy or effort required, with λ representing the tradeoff between gain and transient cost. We will often omit the dependence of V on P and λ for brevity. Example. Consider the Safe Delivery example (Figure 7.1 (Right)). For policy 1, P [1 |= φ] = 0 and so 1 ̸∈ Πmax . Let policy 2 be a cost 1 timestep before stolen or receipt, then g2 = R is the (conditional) gain, J2 = 1 is the (conditional) transient costs, P [2 |= φ] = 1 − p, and V2 = (1 + λR)(1 − p). Problem 1 (Planning with Generative Model/Simulator). Suppose access to a generative model of the true dynamics P from which we can sample transitions s′ ∼ P (s, a) for any state-action pair (s, a) ∈ S × A.2 With probability 1 − δ, for some errors ϵφ , ϵV > 0, find a policy π ∈ Π that simultaneously has the following properties: (i) |P [π |= φ] − P [π ∗ |= φ] | < ϵφ (ii) |Vπ − Vπ∗ | < ϵV . 1 Normalized objectives are not unusual in RL, e.g. in discounted settings, multiplication by (1 − γ) 2 The use of a generative model is increasingly common in RL [72, 126, 6, 198], and is applicable in many settings where such a generative model is readily available as a simulator (e.g., [52]). 73 7.3 Approach End Components & Accepting Maximal End Components Our analysis relies on the idea of an end component: a recurrent, inescapable set of states when restricted to a certain action set. It is a sub-MDP of a larger MDP that is probabilistically closed. Definition 7.3.1. (End Component, EC/MEC/AMEC [16]) Consider MDP (S, A, P, C, d0 , L, S ∗ ). An end component (E, AE ) is a set of states E ⊆ S and acceptable actions AE (s) ⊆ A(s) (where s ∈ E) such that ∀(s, a) ∈ E × AE then P ost(s, a) = {s′ |P (s, a, s′ ) > 0} ⊆ E. Furthermore, (E, AE ) is strongly connected: any two states in E is reachable from one another by means of actions in AE . We say an end component (E, AE ) is maximal (MEC) if it is not contained within a larger end component (E ′ , AE ′ ), i.e., ∄(E ′ , AE ′ ) EC where E ⊆ E ′ , AE (s) ⊆ AE ′ (s) for each s ∈ A. A MEC (E, AE ) is an accepting MEC (AMEC) if it contains an accepting state, ∃s ∈ E s.t. s ∈ S ∗ . High-Level Intuition The description of our approach, LTL Constrained Planning (LCP), in Section 7.3 is rather technical in order to yield theoretical guarantees. We thus first summarize the high-level intuitions. β s*1 A1 s* s*2 1−β A3 A2 s* (b) Example, Infinite Loop (c) Example, Safe Delivery (a) Abstract Diagram Figure 7.2: Product MDP diagrams. (Left) The goal of LTL Constrained Policy Optimization can be reduced to a reachability problem. We want to reach A1 or A2 from s0 and then follow the blue arrows with some distribution. A3 with the blue arrows is a rejecting end component because it does not contain an accepting state s∗ . For β < 1 , the yellow action is not in the allowable action set of A1 because there is a risk of entering A3 , strictly decreasing our probability of LTL satisfaction. (Center) Example for Infinite Loop, Figure 7.1 Left. (Right) Example for Safe Delivery, Figure 7.1 Right. Solution Decomposition. Consider the accepting states s∗1 , s∗2 in Figure 7.2 (Left), which are the states we need to visit infinitely often to satisfy the 74 specification. First, let us identify the accepting maximal end components (AMECs) of s∗1 and s∗2 : the state sets A1 and A2 (resp.) and their corresponding action sets AA1 and AA2 (the blue arrows in A1 and A2 ). Note that these AMECs do not include the yellow action in Figure 7.2 (Left), which has a chance of leaving A1 and getting stuck in A3 . Our solution first runs a transient policy until reaching A1 or A2 , and then switches to a (probability-optimal) recurrent policy that stays within A1 or A2 (resp.) while visiting s∗1 or s∗2 (resp.) infinitely often. A probability-optimal recurrent policy will select actions in AA1 and AA2 to visit s∗1 , s∗2 infinitely often (e.g., the uniform policies with the AMECs (A1 , AA1 ) and (A2 , AA2 )). Finding a transient policy from s0 to A1 , A2 can be viewed as a reachability problem, which we can solve via a Stochastic Shortest Path (SSP) problem and leverage recent literature [198, 110]. Cost Optimality. As stated in (LTL-OPT), the goal is to find a cost-optimal policy within the set of probability-optimal policies. For instance, the uniform policy over AA1 and AA2 (the blue arrows in Figure 7.2 (Left) is probability optimal, but may not be cost optimal. Similarly, the unconstrained costoptimal policy may not be probability optimal. Consider just A1 for the moment. Suppose the cost of the arrows between the white nodes is 4 while the other costs are 7. Then the uniform (probability-optimal) policy in A1 over AA1     1 7+7+4 has cost 12 4+4 + = 5. The gain-optimal policy that deterministically 2 2 3   selects the actions between the white nodes π̃ has cost 4+4 = 4, but is not 2 probability optimal. If we perturb π̃ to make it even slightly stochastic (but still mostly deterministic, i.e η-greedy with η ≈ 0), then it will be arbitrarily close to gain optimality and also recover probability optimality. This is a preferable probability-optimal policy over the uniform policy. Overall Procedure. The high-level procedure is: (i) identify the AMECs (e.g. (A1 , AA1 ), (A2 , A2 )) by filtering out bad actions like the yellow arrow; (ii) find a cost-optimal (optimal gain cost) recurrent policy in each AMEC that visits some s∗ infinitely often; (iii) instantiate an SSP problem that finds a cost-optimal (optimal transient cost) transient policy from s0 to A1 ∪ A2 and avoids A3 ; (iv) return a policy that stitches together the policies from (ii) and (iii). See Section 7.3 for the algorithmic details. We show in Section 7.4 that this solution gives the optimal solution to LTL-OPT. 75 Additional Assumptions and Definitions Perhaps surprisingly, when planning with a simulator (i.e., generative model), even infinite data is insufficient to verify an LTL formula without having a known lower-bound on the lowest nonzero probability of the transition function P [131]. Without this assumption, LTL constrained policy learning is not learnable [224]. We thus begin by assuming a known lower bound on entries in P .3 Assumption 7.1 (Lower Bound). We assume we have access to a lower bound β > 0 on the lowest non-zero probability of the transition function P : 0<β≤ min s,a,s′ ∈S×A×S {P (s, a, s′ )|P (s, a, s′ ) > 0}. (7.2) We assume that all the costs are strictly positive, avoiding zero-cost (or negativecost) cycles that trap a policy. Leveraging cost-perturbations and prior work [198] can remove the assumption. Assumption 7.2 (Bounds on cost function). The minimum cost cmin > 0 is strictly positive. Let D = {(s, a, s′ )} be all the collected samples (s, a, s′ ) while running the ′ )∈D}| algorithm. At any point, Pb (s, a, s′ ) = |{(s,a,s is the empirical frequency |{(s,a)∈D}| ′ of visiting s from (s, a). We introduce an event E and error ψ(n) to quantify uncertainty on Pb (s, a, s′ ) based on current data: n(s, a) = |{(s, a) ∈ D}|. E is based on empirical Bernstein bounds [141], and holds w.p. 1 − δ (Lemma E.2.1). Definition 7.3.2 (High Probability Event). A high probability event E: E = {∀s, a, s′ ∈ S × A × S, ∀n(s, a) > 1 : |(P (s,a, s′ ) − Pb (s, a, s′ ))| ≤ ψsas′ (n) ≤ ψ(n)}, q where ψsas′ (n) ≡ 2P̂ (s, a, s′ )(1 − P̂ (s, a, s′ )))ξ(n) + 73 ξ(n), ψ(n) ≡ 4n2 |S|2 |A| 7 ξ(n), and ξ(n) ≡ log( )/(n − 1). 3 δ q 1 ξ(n) + 2 Remark 7.3.1. For some ρ > 0, if we require |P (s, a, s′ ) − Pb (s, a, s′ )| ≤ ρ then we need n(s, a) = ψ −1 (ρ) samples for state-action pair (s, a). See Lemma E.2.2 for the quantity ψ −1 (ρ). 3 Our assumptions are consistent with the minimal requirements studied by [131]. 76 Definition 7.3.3 (Plausible Transition Function). The set of plausible transition functions is given by P = {P̃ :S × A → ∆(S)| (7.3)   P̃ (s, a, s′ ) = Pb (s, a, s′ ), Pb (s, a, s′ ) ∈ {0, 1}  P̃ (s, a, s′ ) ∈ Pb (s, a, s′ ) ± ψsas′ ∩ [β, 1 − β], otherwise } (7.4) Let P(s, a) ≡ {P (s, a, ·)|P ∈ P} be the possible transition distributions for state-action pair (s, a). We denote Pπ (s, s′ ) = Ea∼π [P (s, a, s′ )] as the Markov chain given dynamics P with policy π, and can be thought of as a |S| × |S| |S| matrix Pπ = {pij }i,j=1 . Main Algorithm: LTL Constrained Planning (LCP) Algorithm 6 LTL Constrained Planning (LCP) Require: Error ϵV > 0, Error ϵφ > 0, Tolerance δ > 0, Lower bound β > 0 (see Assumption 7.1) ′ )∈D}| 1: Globally, track Pb(s, a, s′ ) = |{(s,a,s {Empirical estimate of P } |{(s,a)∈D}| 2: ((A1 , AA1 ), . . . , (Am , AAk )) ← FindAMEC((S, A, Pb)) 3: for i = 1, . . . , k do V 4: Set πi , gi ← PlanRecurrent((Ai , AAi ), ϵ7λ ) {Plan gain-optimal policy πi for Ai } 5: Set π0 ← PlanTransient(((A1 , g1 ), . . . , (Ak , gk )), 2ϵ9V ) {Plan shortest paths policy π0 to ∪ki=1 Ai } 6: return π = ∪ki=0 πi Our approach, LTL Constrained Planning (LCP), has three components, as shown in Algorithm 6 and described below. Recall from Problem 1 that the policy optimization problem LTL-OPT is instantiated over a product MDP (Def. 7.2.1), and that we are given a generative model of the true dynamics P from which we can sample transitions s′ ∼ P (s, a) for any state/action pair. Finding AMECs (FindAMEC). Sampling each state-action pair φFindAMEC = O( β1 ) times (see Prop. E.2.4), by Assumption 7.1, verifies the support of P . We can compute all of the MECs using Algorithm 47 from [16]. Among these MECs, we keep the AMECs, which amounts to checking if the MEC (Ai , AAi ) contains an accepting state s∗ ∈ S ∗ from the given product MDP. PlanRecurrent (PR). To plan in each AMEC (A, AA ) (i.e., find the optimal recurrent policy), we use Alg. 7 with (extended) relative value iteration (VI, Alg. 17 in appendix) using the optimistic Bellman operator LαPR (see Table 7.1, we discuss α in next paragraph). Let πv denote the greedy policy w.r.t. the fixed point v = LαPR v (v is the optimistic value estimate). Using the η-greedy policy, π ≡ (1 − η)πv + ηUnif(AA ) (Alg. 7, Line 7), together with Pπ , makes A 77 Algorithm 7 PlanRecurrent (PR) Require: AMEC (A, AA ), error ϵPR > 0 1: Set ρ ← 2ψ(φFindAMEC (β)) 2: repeat 3: Set ρ ← ρ2 4: Sample ψ −1 (ρ) times ∀(s, a) ∈ A × AA 5: v ′ , v, P̃ ← VI(LαPR , dPR , ϵL PR ) ϵPR (1−∆(P̃ )) 6: until ρ > 3|A|cmax {ρ ∼ ∥P − P̃ ∥−1 1 } {v ′ = LαPR v} {∥P − P̃ ∥1 small} 7: Set policy π ← η-greedy policy w.r.t. v ′ 8: Set gain gπ ← 12 (max(v ′ − v) + min(v ′ − v)) 9: return π, gπ recurrent: s∗ ∈ A is visited infinitely often and P [π |= φ|s0 ∈ A] = 1. Since η can be arbitrarily small (Lemma E.2.7), then gπ ≈ gπv and π is both cost and probability optimal. As intuited in Section 7.3, π has full support over AA but is nearly deterministic.4 VI in Line 5 of Alg. 7 is an iterative procedure (Alg. 17 in appendix), and terminates via dPR < ϵLPR (Table 7.1). Convergence of extended VI is guaranteed [166, 94, 66], so long as the dynamics, P̃ = arg minp∈P(s,a) pT v, achieving the inner minimization of LαPR are aperiodic — hence the aperiodicity transform α ∈ (0, 1) in LαPR [166]. Computing P̃ can be done efficiently [94] (Alg. 18 in appendix). For stability, we shift each entry of vn by the value of the first entry vn (0) [21]. Alg. 7 returns the average gain cost gπ of policy π when we have enough samples   (1−∆(P̃π )) for each state-action pair in (A, AA ) to verify that n > ψ −1 ϵPR3|A|c where max P 1 ∆(P̃π ) = 2 maxij k |p̃ik − p̃jk |. Here, ∆(P̃π ) is an easily computable measure on the ergodicity of the Markov chain P̃π [41]. We track ψ(n) (recall Def. 7.3.2) via a variable ρ and sample ψ −1 (ρ) ≈ ρ12 (see Lemma E.2.2) samples from each state-action pair in (A, AA ) (Alg. 7, Line 4). We halve ρ each iteration (Alg. 7, Line 3) and convergence is guaranteed because ρ will never fall below some ¯ A) (1−∆ unknown constant ϵPR6|A|c (see Lemma E.2.8); the halving trick is required max ¯ A is unknown a priori. because ∆ Proposition 7.3.2 (PR Convergence & Correctness, Informal). Let πA be the gain-optimal policy in AMEC (A, A). Algorithm 7 terminates after at most 4 Typically, RL settings admit a fully deterministic optimal policy, but for LTL constrained policy optimization the optimal policy may not be deterministic (although can be very nearly so). See Cost Optimality in Section 7.3. Table 7.1: Subroutine Operators and Parameters for Value Iteration. Op/Param 78 Description
Lα mina∈AA (s) C(s, a) + α minp∈P(s,a) pT v + (1 − α)v(s) ∀s ∈ A PR v(s) 2ϵPR L dPR (vn+1 , vn ) < ϵPR  maxs∈A  (vn+1 (s) − vn (s)) − mins∈A (vn+1 (s) −
vn (s)) < 3 min mina∈AA (s) C(s, a) + minp∈P(s,a) pT v , V̄ /ϵφ , s ∈ S \ ∪ki=1 Ai LPT v(s) λgi , s ∈ Ai dPT (vn+1 , vn ) < ϵL ∥v − v ∥ < c ϵ ϵ /(4 V̄ ) n+1 n 1 min PT φ PT Algorithm 8 PlanTransient (PT) Require: States & gains: {(Ai , gi )}ki=1 , err. ϵPT > 0 1: Set VT (s) = λgi for s ∈ Ai 2: Sample φPT times ∀(s, a) ∈ (S \ ∪Ai ) × A 3: v ′ , v, P̃ ← VI(LPT , dPT , ϵL PT , VT ) 4: Set π ←greedy policy w.r.t v ′ 5: return π   {Terminal costs} {v ′ = LPT v} 2 2 cmax max log2 ϵPR6|A|c repeats, and collects at most n = Õ( ϵ2|A| ¯ A) ¯ 2 ) samples for (1−∆ PR (1−∆A ) each (s, a) ∈ (A, AA ). The η-greedy policy π w.r.t. v ′ (Alg. 7, Line 5) is gain optimal and probability optimal: |gπ − gπA | < ϵPR , P [π |= φ|s0 ∈ A] = 1. PlanTransient (PT). This is the stochastic shortest path (SSP) reduction step that finds a policy from the initial state s0 to the AMECs (Alg. 8). The main algorithmic tool used by PlanTransient is similar to that of PlanRecurrent: it also uses extended value iteration (VI, Alg. 17 in appendix) but with a different optimistic Bellman operator LPT (Table 7.1), and then returns a (fully deterministic) greedy policy w.r.t. the resulting optimistic value v (Alg. 8, Line 4). LPT is used to calculate the highest probability, lowest cost path to the AMECs (Alg. 8, Line 3). Since rejecting end components might exist (see A3 from Figure 7.2 (Left)), a trajectory may end up stuck and accumulate cost indefinitely, and so we must bound ∥v∥∞ < V̄ /ϵφ to prevent blow up. In Prop. E.2.13, we show how to select V̄ such that π will reach the target states (in this case, the AMECs), first with high prob and then with lowest cost. The existence of such a bound on ∥v∥∞ was shown to exist, without construction, in [110]. In practice, choosing a large V̄ is enough. The terminal costs VT (Alg. 8, Line 1) together with Bellman equation LPT has P value function Ṽπ ≈ p(Jπ + p1 ki=1 pi gπi ) + (1 − p)V̄ /ϵφ ≈ Vπ , relating to Vπ (LTL-OPT); see Section E.1. Here, pi = P [π reaches Ai ] ≡ Eτ ∼TPπ [1∃s∈τ s.t s∈Ai ] 79 and p = i=1 pi . VI converges when dPT < ϵPT (see Table 7.1). Convergence of extended VI for SSP is guaranteed [198, 110]. The number of samples required  Pk for each state-action pair (s, a) ∈ (S \ ∪Ai ) × A is φPT = ψ −1 cmin ϵPT ϵ2φ 14|S\∪ki=1 Ai |V̄ 2 . Proposition 7.3.3 (PlanTransient Convergence & Correctness, Informal). Denote the cost- and prob-optimal policy as π ′ . After collecting at most n = |S\∪k A |2 V̄ 4 Õ( c2i=1ϵ2 iϵ4 ) samples for each (s, a) ∈ (S \ ∪ki=1 Ai ) × A, the greedy policy π min PT φ w.r.t. v ′ (Alg. 8, Line 3) is both cost and probability optimal: ∥Ṽπ − Ṽπ′ ∥ < ϵPT , 7.4 h i h i |P π reaches ∪ki=1 Ai − P π ′ reaches ∪ki=1 Ai | ≤ ϵφ . End-To-End Guarantees The number of samples necessary to guarantee an (ϵV , ϵφ , δ)-PAC approximation to the cost-optimal and probability-optimal policy relies factors: β (lower bound on the min. non-zero transition probability of P ), {cmin , cmax } (bounds on the ¯ A (worst-case coefficient of ergodicity for EC (Ai , AA )), V̄ cost function C), ∆ i i (upper bound on the value function), and λ (tradeoff factor). Theorem 7.4.1 (Sample Complexity). Under the event E, Assumption 7.1 and 7.2, after !! k X 1 1 |S|2 V̄ 4 |Ai |2 c2max 2 n = Õ + +λ ¯ 2 β ϵ2V c2min ϵ4φ i=1 (1 − ∆Ai ) samples5 are collected from each state-action pair, the policy π returned by Algorithm 6 is, with probability 1 − δ, simultaneously ϵV -cost optimal and ϵφ -probability optimal, satisfying: (i) |P [π |= φ] − P [π ∗ |= φ] | ≤ ϵφ (ii) ∥Vπ − Vπ∗ ∥∞ < ϵV . (7.5) With a sufficiently large λ (which may not be verifiable in practice), π is also gain optimal. Corollary 7.4.2 (Gain (Average Cost) Optimality). There exists λ∗ > 0 s.t. for λ > λ∗ , the policy π returned by Alg. 6 satisfies (7.5), gπ = arg minπ′ ∈Πmax gπ′ , and is probability and gain optimal. The high-level structure of our analysis follows the algorithm structure in Section 7.3, via composing the constituent guarantees. To complete the analysis, we 5 log(2δ) The lower bound relating to β from [131] is Ω( log(1−β) ) whereas ours is Õ( β1 ). We conjecture that Ω̃( β1 ) samples are required. See Appendix Section E.3. 1.0 0.8 60 40 20 20 30 40 50 60 Total Number of Samples (× 103 ) 0.6 0.4 0.2 0.0 200 190 180 170 160 150 LCP (Ours) LCRL (shaped) Max LCRL Better Better 80 Mountain Car 1.0 0.8 Better Better 0.6 0.4 0.2 0.0 100 Timesteps Pacman Prob of Satisfaction 80 50 100 150 200 Total Number of Samples (× 103 ) Figure 7.3: Results. (Left Column) Pacman. φ is to eventually collect food and always avoid the ghost. We let the system run for a maximum of 100 timesteps. (Right Column) Discretized Mountain Car (MC). φ is to eventually reach the flag. develop some technical tools which may be of independent interest, including a gain simulation Lemma E.2.8 and an η-greedy optimality Lemma E.2.7. For ease of exposition, we also ignore paths between AMECs (see Appendix E.4). 7.5 Empirical Analysis We perform experiments in two domains: (1) Pacman domain where an agent finds food and indefinitely avoids a ghost and (2) discretized version of mountain car (MC) [29], where the agent must reach the flag. Our goal is to understand whether: (i) our LCP approach (Alg.6) produces competitive polices; (ii) LCP can work in continuous state spaces through discretization; (iii) LCP can enjoy efficient sample complexity in practice. For a baseline, we use Logically Constrained RL (LCRL, [83]), which is a Q-learning approach to LTL-constrained PO in unknown MDPs. We also do heavy cost shaping to LCRL as another baseline. See App E.5 for more details and results. Results spec in Figure 7.3 (Left) approaches 1 much faster than the two baselines. The returned policy collects the food quickly and then stays close, but avoids, the ghost. Any policy that avoids the ghost is equally good, as we have not incentivized it to stay far away. LCRL redefines cost as 1 if the LTL is solved and 0 otherwise, which is too sparse and learning suffers. Indeed, shaped LCRL performs better than straight LCRL. 81 Performance in continuous state space? Similarly, the probability of satisfying the LTL spec in Figure 7.3 (Right) goes up to 1. However, here the LCRL (shaped) baseline performs relatively well as it is being given “breadcrumbs” for how to solve the task. Our algorithm performs well without needing any cost shaping. Standard LCRL fails to learn. This experiment demonstrates that LCP can be used even in discretized continuous settings. Sample Complexity? Our theory is quite conservative w.r.t. empirical performance. In Pacman (Figure 7.3, Left), Thm. 7.4.1 suggests ≈ 350 samples per (s, a) pair just to calculate the AMECs. Empirically, LCP finds a good policy after 11 samples per (s, a) pair (∼ 66k/6k samples/pair). Other Considerations. One of the strengths and potential drawbacks of LTL is its specificity. If a φ, for a truly infinite horizon problem, is to “eventually” do something, then accomplishing the task quickly is not required. As a finite horizon problem, in MC (Fig. 7.3, Right) SSP finds the fastest path to the goal. In contrast, since any stochastic policy with full support will “eventually” work, the policy returned by LCP for Fig 7.1 (Left) (Fig. 7.2 Center, & App Fig. E.4) may take exponential time to complete a single loop. Two straightforward ways to address this issue are (a) including explicit time constraints in φ, and (b) cost shaping to prefer policies reaching some s∗ quickly and repeatedly. Unlike standard cost-shaping, φ satisfaction is still guaranteed since the cost is decoupled from φ. 7.6 Related Work Constrained Policy Optimization. One attempt at simplifying cost functions is to split the desired behaviors from the required behaviors. The desired behaviors remain as part of the cost function while the required behaviors are treated as constraints. Recent interest in constrained policy optimization within the RL community has been related to the constrained Markov Decision Process (CMDP) framework [10, 123, 4, 142]. This framework enables clean methods and guarantees, but enforces expected constraint violations rather than absolute constraint violations. Setting and interpreting constraint thresholds can be very challenging, and inappropriate in safety-critical problems [121]. LTL + RL. Recently, LTL-constrained policy optimization has been developed as an alternative to CMDPs [131]. Unlike CMPDs, the entire task is encoded 82 into an LTL expression and is treated as the constraint. Q-learning variants when dynamics are unknown and Linear Programming methods when dynamics are known are common solution concepts [176, 83, 27, 33, 50]. The Q-learning approaches rely on proper, unknowable tuning of discount factor for their guarantees. Theoretically oriented works include [67, 221]. While providing PAC-style guarantees, the assumptions made in these works rely on unknowable policy-environment interaction properties. We make no such assumptions here. Another solution technique is employing reward machines [205, 34, 210] or high-level specifications that can be translated into reward machines [99]. These works are generally empirical and handle finite or repeated finite problems (episodic problems at test time); they can only handle a smaller set of LTL expressions, specifically regular expressions. Our work handles ω-regular expressions, subsuming regular expressions and requires a nontrivial leap, algorithmically and theoretically, to access the broader set of allowable expressions. Many problems are ω-regular problems, but not regular, such as liveness (something good will happen eventually) and safety (nothing bad will happen forever). The works that attempt to handle full LTL expressibility redefine reward as 1 if the LTL is solved and 0 otherwise; the cost function of the MDP is entirely ignored. Verification and Planning. As an alternative to our approach, one might consider LTL satisfaction verification and extend it to an optimization technique by checking every policy (which will naively take an exponential amount of samples to verify a single policy [28, 13]). Many verification approaches exist [114, 16, 8, 226, 117, 90] and among the ones that do not assume known dynamics, the verification guarantees rely on quantities as difficult to calculate as the original verification problem itself [13]. 7.7 Discussion We have presented a novel algorithm, LCP, for policy optimization under LTL constraints in an unknown environment. We formally guarantee that the policy returned by LCP simultaneously has minimal cost with respect to the MDP cost function and maximal probability of LTL satisfaction. Our experiments verify that our policies are competitive and our sample estimates conservative. The assumptions we make are strong, but to the best of our knowledge, are the most relaxed amongst tractable model-based algorithms proposed for this 83 space. Model-free algorithms (Q-learning) have less stringent assumptions but do not come with the kind of guarantees that our work has and largely ignore the cost function, solving only part of the problem. An interesting future direction would be to extend our work to continuous state and action spaces and settings with function approximation. 84 Chapter 8 LTL-BASED GUARANTEES IN CONTINUOUS DOMAINS Linear temporal logic (LTL) offers a simplified way of specifying tasks for policy optimization that may otherwise be difficult to describe with scalar reward functions. However, the standard RL framework can be too myopic to find maximally LTL satisfying policies. This chapter makes two contributions. First, we develop a new value-function based proxy, using a technique we call eventual discounting, under which one can find policies that satisfy the LTL specification with highest achievable probability. Second, we develop a new experience replay method for generating off-policy data from on-policy rollouts via counterfactual reasoning on different ways of satisfying the LTL specification. Our experiments, conducted in both discrete and continuous state-action spaces, confirm the effectiveness of our counterfactual experience replay approach. We focus on model-free policy learning of an LTL specified objective from online interaction with the environment. We make two technical contributions. First, we reformulate the RL problem with a modified value-function proxy using a technique we call eventual discounting. The key idea is to account for the fact that optimally satisfying the LTL specification may not depend on the length of time it takes to satisfy it (e.g., “eventually always reach the goal”). We prove in Section 8.2 that the optimal policy under eventual discounting maximizes the probability of satisfying the LTL specification. Second, we develop an experience replay method to address the reward sparsity issue. Namely, any LTL formula can be converted to a fully known specialized finite state automaton from which we can generate multiple counterfactual trajectories from a single on-policy trajectory. We call this method LTL-guided counterfactual experience replay. We empirically validate the performance gains of our counterfactual experience replay approach using both finite state/action spaces as well as continuous state/action spaces using both Q-learning and Policy Gradient approaches. 85 ⊥ 0  A 1 ¬y ⊥ 2 y 1 y ¬y 0 ¬r r 0 B α b b 2 ¬r 3 b 1−α ⊥ Figure 8.1: Examples. First: Illustration of the Flatworld environment. The agent is a green dot and there are 3 zones: yellow, blue and red. Second: ∗ LDBA B for “F Gy”. S B = {1}, denoted by green circle. The initial state is ∗ b−1 = 0. Third: LDBA B for “GF (y & XF r) & G¬b”. S B = {1}, denoted by green circle. The initial state is b−1 = 1. Fourth: For this example, an agent P i B∗ ∗ starting in state 0 and solving arg maxπ∈Π Eτ ∼TPπ [ ∞ i=0 γ 1{bi ∈S B } ] where S is illustrated as the green circles would choose to take action B with probability 1 if α ∈ (1/2, 1) for any γ ∈ [0, 1]. Such a policy has P [π |= φ] = α < 1. However the probability optimal policy deterministically takes action A, with P [π ∗ |= φ] = 1. This illustrates catastrophic myopic behavior. 8.1 Problem Formulation For an introduction to the background for LTL, see Section 2.3. From there, recall a few key definitions. Let Z = S M × S B . Let Π : Z × A → ∆([0, 1]) be a (stochastic) policy class over the product space of the MDP and the LDBA (defined below), where A((s, b)) = AM (s) ∪ AB (b), to account for jump transitions in B. Synchronizing the MDP with the LDBA. For any (s, b) ∈ Z, a policy π ∈ Π is able to select an action in AM (s) or an action in AB (b), if available. We can therefore generate a trajectory as the sequence τ = (s0 , b0 , a0 , s1 , b0 , a1 , . . .) under a new probabilistic transition relation given by P (s′ , b′ |s, b, a) =    P M (s, a, s′ )    1,     0, a ∈ AM (s), b′ ∈ P B (b, L(s′ )) a ∈ AB (b), b′ ∈ P B (b, a), s = s′ . (8.1) otherwise Let the LDBA projection of τ be the subsequence τB = (b0 , b1 , . . .). Elements of τB can be thought of as tracking an agent’s LTL specification satisfaction: Definition 8.1.1 (Run Satisfaction, τ |= φ). We say a trajectory satisfies φ if ∗ B accepts τB , which happens if ∃b ∈ τB infinitely often with b ∈ S B . Let TPπ = Ez∼dM [TPπ (z)] be the distribution over all possible trajectories 0 ×{b−1 } P starting from any initial state z ∈ dM 0 × {b−1 } where Tπ (z) is the (conditional) 86 distribution over all possible trajectories starting from z ∈ Z generated by π under relation P (given in (8.1)). The probability of LTL satisfaction results from counting how many of the trajectories satisfy the LTL specification: Definition 8.1.2 (State Satisfaction, z |= φ). Pπ [z |= φ] = Eτ ∼TPπ (z) [1{τ |=φ} ] = Eτ ∼TPπ [1{τ |=φ} |z0 = z]. Definition 8.1.3 (Policy Satisfaction, π |= φ). P[π |= φ] = Eτ ∼TPπ [1{τ |=φ} ] where 1X is the indicator for X. Ideally we would like to find a policy with highest probability of LTL specification satisfaction: one that generates the most number of LTL-satisfying runs. Formally, π ∗ ∈ arg max P[π |= φ]. (8.2) π∈Π We note that Eq (8.2) is the standard starting point for formulating policy optimization for LTL satisfaction [224, 27, 32, 83, 84, 215]. 8.2 RL-Friendly Form: Eventual Discounting Unfortunately, the maximization problem in Eq (8.2) is not easily optimized since we dont have a direct signal on P[π |= φ]. Without any additional assumptions (such as structured knowledge of the MDP), any finite subsequence can only give evidence on whether τ |= φ but not a concrete proof. Eventual Discounting. To address the above issue, we develop a modified value-function based surrogate as follows. Given a trajectory τ = (s0 , b0 , a0 , . . .), ∗ ∗ we keep track of how often bi ∈ S B and incentivize an agent to visit S B as many times as possible. In particular, under eventual discounting, the value ∗ function will give the agent a reward of 1 when in a state bi ∈ S B and not ∗ discount length of time between visits to S B . Formally, we will be seeking πγ∗ ∈ arg max Eτ ∼TPπ [ π∈Π ∞ X Γi 1{bi ∈S B∗ } ] (≡ Vπγ ), (8.3) i=0 where Γ0 = 1 and Γi = i−1 Y t=0 γ(bt ),   γ, γ(bt ) =  1, bt ∈ S B ∗ otherwise . (8.4) Intuition for Γi . At first glance setting Γi = γ i to be the traditional RL exponential discount rate would seem reasonable. Unfortunately, ∄γ ∈ [0, 1] with Γi = γ i that avoids catastrophic myopic behavior. In particular, take 87 Figure 8.1 (Fourth). The agent starts in state 0 and only has two actions A and B. Taking action A transitions directly to an accepting state from which point the accepting state is visited every 2 steps. On the other hand, action B transitions to an accepting state with probability α and a sink state with probability 1 − α. The accepting state reached by action B is revisited every step. Suppose β = π(A) = 1 − π(B) then we can calculate: ∞ X β (1 − β)α Eτ ∼TPπ [ γ i 1{bi ∈S B∗ } ] = . (8.5) + 2 1−γ 1−γ i=0 For α > 1/2, the optimal choice β is β = 0 implying that P (π |= φ) = α. When α ∈ (1/2, 1) then this implies that π is not probability optimal. Indeed, P (π |= φ) = α < 1 when β = 0 but P (π ∗ |= φ) = 1 by selecting β = 1. The intuition here, which can be formalized by taking γ → 1, is that the average reward for taking action A is 12 while the average reward for taking action B is 1 with probability α, which is worth the risk for large α > 1/2. To avoid this myopic behavior, we must avoid discriminating between return times between good states. The number steps (on average) it takes to return to ∗ S B is irrelevant: we only require that the system does return. For this reason we do not count time (hence γ = 1) in our definition of Γi when the system is ∗ not in S B . We call this eventual discounting. Analysis of πγ∗ In this section we analyze how the probability of πγ∗ satisfying φ compares to that of the best possible one π ∗ . ∗ Let the set O(τ ) = {i : bi ∈ S B } denote the occurences (time steps) when a good state is reached. This quantity is natural since |O(τ )| = ∞ if and only if τ |= φ. Lemma 8.2.1. For any π ∈ Π and γ ∈ (0, 1), we have 1 |(1 − γ)Vπγ − P[π |= φ]| ≤ log( )Oπ , γ " # where Oπ = Eτ ∼TPπ |O(τ )| τ ̸|= φ is the expected number of visits to an accepting state for the trajectories that do not satisfy φ. 88 Proof. Fix some state z = (s, b) ∈ Z. Vπγ (z) = Eτ ∼TPπ [ ∞ X Γi 1{bi ∈S B∗ } |z0 = z] i=0 |O(τ )|  = Eτ ∼TPπ  X  γ j |z0 = z  . j=0 Using the fact that Pk j=0 γ j 1−γ k = 1−γ , we have " Vπγ (z) = Eτ ∼TPπ 1 − γ |O(τ )| τ |=φ Pπ [z |= φ] 1 − γ z0 =z # " + Eτ ∼TPπ 1 − γ |O(τ )| τ ̸|=φ Pπ [z ̸|= φ]. 1 − γ z0 =z # (8.6) Since |O(τ )| = ∞ for any τ |= φ, " # 1 − γ |O(τ )| τ |=φ 1 Eτ ∼TPπ , z0 =z = 1−γ 1−γ together with Pπ [z ̸|= φ] ≥ 0 implies 1 Vπγ (z) ≥ Pπ [z |= φ]. 1−γ Taking the expectation over initial states we have 1 Vπγ ≥ P[π |= φ]. 1−γ (8.7) (8.8) (8.9) " # Now we find an upper bound. Let Mπ (t) = Eτ ∼TPπ e t|O(τ )| τ ̸|= φ . Starting again with Eq (8.6) and using Eq (8.7), we have # " 1 − Eτ ∼TPπ elog(γ)|O(τ )| zτ0̸|=φ =z P [z |= φ] π + , (8.10) Vπγ (z) ≤ 1−γ 1−γ where we have used that Pπ [z ̸|= φ] ≤ 1 for any z ∈ Z. Taking the expectation with respect to the initial state distribution then we have (1 − γ)Vπγ ≤ P[π |= φ] + 1 − Mπ (log(γ)) (8.11) In particular, Mπ (t) is convex and therefore it lies above its tangents: " Mπ (t) ≥ Mπ (0) + tMπ′ (0) = 1 + tEτ ∼TPπ # |O(τ )| τ ̸|= φ = 1 + tOπ . Plugging this inequality into Eq (8.11), together with Eq (8.9), 1 P[π |= φ] ≤ (1 − γ)Vπγ ≤ P[π |= φ] + log( )Oπ . (8.12) γ Subtracting P[π |= φ] from both sides and taking the absolute value completes the proof. 89 Theorem 8.2.2. (Non-asymptotic guarantee) For any γ ∈ (0, 1), 1 sup P[π |= φ] − P[πγ∗ |= φ] ≤ 2 log( ) sup Oπ , γ π∈Π π∈Π " (8.13) # where Oπ = Eτ ∼TPπ |O(τ )| τ ̸|= φ . Proof. Consider any sequence {πi }∞ i=1 such that P[πi |= φ] → supπ P[π |= φ] as i → ∞. Then we have for any πi , P[πi |= φ] − P[πγ∗ |= φ] = P[πi |= φ] − (1 − γ)Vπγi + (1 − γ)Vπγi − (1 − γ)Vπγγ∗ + (1 − γ)Vπγγ∗ − P[πγ∗ |= φ] (a) ≤ |P[πi |= φ] − (1 − γ)Vπγi | + |P[πγ∗ |= φ] − (1 − γ)Vπγγ∗ | 1 ≤ log( )(Oπi + Oπγ∗ ) γ (c) 1 ≤ 2 log( ) sup Oπ , γ π∈Π where (a) is triangle inequality together with removing the term (1 − γ)Vπγi − (1 − γ)Vπγγ∗ since it is nonpositive by definition of πγ∗ , (b) is an application of Lemma 8.2.1, and (c) is a supremum over all policies. Taking the limit on both sides as i → ∞ completes the proof. (b) Corollary 8.2.3. If the number of policies in Π is finite then supπ∈Π Oπ = m < ∞ is attained, is a finite constant and 1 sup P[π |= φ] − P[πγ∗ |= φ] ≤ 2m log( ). γ π∈Π Corollary 8.2.4. In the case that S M and AM are finite, then Z and A are finite. It is known that optimal policies are deterministic [166] and therefore there we need only consider deterministic policies, for which there are a finite number. Thus supπ∈Π Oπ = m < ∞ is attained, is a finite constant and 1 sup P[π |= φ] − P[πγ∗ |= φ] ≤ 2m log( ). γ π∈Π Interpretation Theorem 8.2.2 relies on the quantity supπ∈Π Oπ to be finite for the bound to have meaning. In fact, we need only make requirements on Mπ (log(γ)) but the requirements are more easily understood on Oπ . As an aside, Mπ (log(γ)) 90 can be interpreted as the moment generating function of the random variable ∗ which is the number of visits to S B . Instead we consider the equally natural quantity Oπ . Oπ is the (average) number of times that a good state is visited by a trajectory that does not satisfy the specification. Ideally, this number would be small and it would be easy to discriminate against good and bad policies. The bad news. In the case that Π is an infinite class, conditions for ensuring supπ∈Π Oπ is finite is nontrivial and is dependent on the landscape of the transition function P of the MDP and Π. Let us suppose supπ∈Π Oπ is infinite. This means there are policies that induce ∗ bad trajectories that eventually fail to reach S B , but along the way visited ∗ S B an arbitrarily large (but finite) number of times. In other words, they are policies that are indistinguishable from actual probability-optimal policies until the heat death of the universe. For example, let us consider the specification in Figure 8.1 (Third), given by “indefinitely cycle between red and yellow while avoiding blue”. A good-looking bad policy is one that accomplishes the task frequently but, amongst the times that it fails, it would cycle between red and yellow many times before failing. supπ∈Π Oπ being infinite means that there are policies that will cycle arbitrarily many times before failing. The good news. Corollary 8.2.4 reveals that discretization suffices to generate probability optimal policies, with suboptimality shrinking at a rate of log( γ1 ). This suggests that compactness of P and Π and continuity of P may very well be enough but we leave these conditions for future work. Finally, since all computers deal with finite precision, the number of policies is finite and therefore Corollary 8.2.3 similarly applies. 8.3 LTL Counterfactual Experience Replay One can optimize the formulation in Eq (8.3) using any Q-learning or policy gradient approach, as seen in Algorithm 9 (Line 4). However, doing so is challenging since it suffers from reward sparsity: the agent only receives a signal if it reaches a good state. We combat reward sparsity by exploiting the LDBA: P B is completely known. By knowing P B , we can generate multiple off-policy trajectories from a single on-policy trajectory by modifying which stats in the LDBA we start in, which 91 Algorithm 9 Learning with LCER Require: Maximum horizon T . Replay buffer D = {}. 1: for k = 1, 2, . . . do 2: Run πk−1 in the MDP for T timesteps and collect trajectory τ = (s0 , b0 , a0 , . . . , sT −1 , bT −1 , aT −1 , sT , bT ) 3: Dk ← LCER(Dk−1 , τ ) 4: πk ← Update(πk−1 , Dk ) {Q-learn/Policy grad.} Algorithm 10 LCER for Q-learning Require: Dataset D. Trajectory τ of length T . 1: for (st , at , st+1 ) ∈ τ do 2: for b ∈ S B do 3: Set b̃ ← P B (b, LM (st+1 )) 4: D ← D ∪ (st , b, at , 1b̃∈B ∗ , st+1 , b̃) 5: for ϵ ∈ AB (s) do 6: Set b̃ ← P B (b, ϵ) 7: D ← D ∪ (st , b, ϵ, 1b̃∈B ∗ , st , b̃) 8: return D notably does not require any access to the MDP transition function P M . We call this approach LTL-guided Counterfactual Experience Replay, LCER (Algorithm 9, Line 3), as it is a modification of standard experience replay [130, 143, 144] to include counterfactual experiences elsewhere in the LDBA. LCER is most simply understood through Q-learning, and needs careful modification for policy gradient methods. Q-learning with LCER. See Algorithm 10 for a synopsis of LCER for Q-learning. Regardless of whatever state s ∈ S M the agent is in, we can pretend that the agent is in any b ∈ S B . Then for any action the agent takes we can store experience tuples: {(s, b, a, r, s′ , b̃′ ) | ∀b ∈ S B }, (8.14) where b̃′ = P B (b, LM (s′ )) is the transition that would have occurred from observing labelled state L(s′ ) in state (s, b) and r = 1b̃′ ∈B ∗ . Furthermore we can add all jump transitions: {(s, b, ϵ, r, s, b̃′ ) | ∀b ∈ S B , ∀ϵ ∈ AB (b)}, (8.15) since jumps also do not affect the MDP. Notice when we add the jumps that s′ = s, since only the LDBA state shifts in a jump. Policy Gradient with LCER. See Algorithm 11 for a summary of LCER for policy gradient. For policy gradient, unlike Q-learning, it is necessary to P calculate future reward-to-go: Rk (τ ) = Ti=k Γi 1{bi ∈S B∗ } . Thus, we have to 92 Algorithm 11 LCER for Policy Gradient Require: Dataset D. Trajectory τ of length T . 1: Set T̃0 ← T̃ (τ ) 2: for k = 1, . . . , T − 1 do 3: T̃k ← E(T̃k−1 ) 4: if T̃k == T̃k−1 then 5: Set T̃T −1 ← T̃k 6: break 7: Set D ← D ∪ T̃T −1 8: return D generate entire trajectories that are consistent with P B rather than independent transition tuples as in Eq (8.14). We will show how to generate all feasible trajectories. Consider a trajectory τ = (s0 , b0 , a0 , . . . , sT , bT ) was collected. Let us remove jump transitions (si , bi , ai ) where ai ∈ AB (bi ) and consider the projection of the trajectory to the MDP τM = (s0 , s1 , . . . , sT ). We should only have control over the initial LDBA state b0 as all other automaton states (b1 , . . . , bT ) in a trajectory sequence are determined by τM and bi+1 = P B (bi , LM (si )). Therefore we add T̃ (τ ) = {(s0 , b̃0 , a0 , . . . , sT , b̃T ) | ∀b̃0 ∈ S B , b̃i = P B (b̃i−1 , LM (si ))}, where only the LDBA states are different between the trajectories. Now we handle jump transitions. Consider some τ̃ ∈ T̃ (τ ). Recall, a jump transition can occur whenever AB (b̃i ) is non-empty. This involves adding a trajectory that is identical to τ̃ all the way until the jump occurs. The jump occurs and then the same action sequence and MDP state sequence follows but with different LDBA states. Specifically, suppose b˜i had an available jump transitions, ϵ ∈ AB (b̃i ). Then: τ̃i,ϵ = (s0 , b̃′i , a0 , . . . , si , b̃′i , ϵ, si , b̃′i+1 , ai , . . . , sT , b̃′T ), (8.16) where b̃′k = b̃i for k ≤ i and b̃′k = P B (b̃′k−1 , LM (sk )) otherwise. ′ We have to add all possible τ̃i,ϵ that exist. Let E be the operator that adds jumps to existing sequences: E(T̃ (τ )) = T̃ (τ ) ∪ {τ̃i,ϵ from Eq (8.16)| ∀τ̃ ∈ T̃ (τ ), ∃bi ∈ τ̃ s.t. ∃ϵ ∈ AB (bi )}. (8.17) We can only apply E(E(. . . (E(T̃ (τ ))))) at most T times since the original length of τ is T . 93 Remark 8.3.1. The length of τ has to be sufficiently large to make sure the ∗ LDBA has opportunity to reach S B . A sufficient condition is T ≥ |{b|AB (b) ̸= ∅}|, the number of LDBA states with jump transitions. It is possible to constructively generate feasible trajectories during the rollout of a policy rather than after-the-fact; see Appendix F.2. Minecraft Pacman ϕ = GF ( x &XF y ) & G¬r ϕ = F( Oscillate yellow & blue, avoid red Get the food, avoid the ghost Flatworld ϕ = FG ) & G¬ Stabilize in yellow Flatworld Carlo ϕ = GF ( x &XF y ) & G¬r ϕ = GF ( x1 &XF y2 ) & G¬r Oscillate yellow & red, avoid blue Drive in a circle, never crash 1 Discounted Reward Discrete Actions 80 2.5 2.0 60 1.5 1.0 0.5 Q-Learn (LCER) Q-Learn 0.0 2 20 Cont. Actions Q-Learn (LCER) Q-Learn PPO (LCER) PPO 1 0 2 4 6 8 10 Training Steps (x10000) Discrete Actions 5 4 3 2 Cont. Actions Q-Learn (LCER) Q-Learn PPO (LCER) PPO .6 .4 PPO (LCER) PPO .2 1 0 0 0 1 2 3 4 5 Training Steps (x1000) 4 3 40 Q-Learn (LCER) Q-Learn 5 2 0 1 2 3 4 5 Training Steps (x1000) 0 0 0 2.5 5 7.5 10 Training Steps (x1000) 0 5 10 15 20 25 Training Steps (x1000) Figure 8.2: Results. Each column is an environment and a LTL formula we’d like an agent to satisfy. The environment and a trajectory from the final policy is illustrated in the center of the column (except for Pacman, which is the initial state). The learning curves at the bottom of each column show that adding off-policy data using LCER has strong benefits for empirical performance. First Column: Minecraft, where an agent should visit the yellow and blue areas while avoiding the red. The final policy is illustrated via blue dots. Second Column: Pacman, where an agent should collect the food while avoiding a ghost. Third Column: Flatword, where an agent should eventually stabilize in the yellow region. When the actions are discrete we use Q-learning, when the actions are continuous we use PPO. Fourth Column: Same as the third column except an agent should oscillate between the yellow and red regions while avoiding the blue. Fifth Column: Carlo, where an agent should drive in a circle without crashing by visiting the blue regions labelled 1 and 2 indefinitely. 8.4 Experiments We perform experiments in four domains with varying LTL formulas, state spaces, action spaces, and environment stochasticity summarized in the following section. Our aim is to answer the following two questions: (1) Can we achieve policies that behave the way we expect an LTL-satisfying policy to behave? (2) How does LCER impact the performance of learning. 94 Environment Details Minecraft The Minecraft environment is a 10 × 10 deterministic gridworld with 5 available actions: left, right, up, down, nothing. The agent, given by a red triangle starts in the cell (9, 2). The environment, as well as the final behavior of the agent (given by blue dots) can be seen in Figure 8.2 (First). Pacman The Pacman environment is a 5 × 8 deterministic gridworld with 5 available actions: left, right, up, down, nothing. The agent, given by a red triangle starts in the cell (0, 3). The ghost chases the agent with probability 0.8 and takes a random action with probability 0.2, for this reason the environment is stochastic. The starting position of the environment can be seen in Figure 8.2 (Second). Flatworld The Flatworld environment (seen in Figure 8.2 Third and Fourth) is a two dimensional continuous world. The agent (given by a green dot) starts at (−1, −1). The dynamics of the world are given by x′ = x + a/10 where both x ∈ R2 and a ∈ [0, 1]2 . We also allow the action space to be discrete by letting there be 5 actions (right, up, left, down, nothing) where the agent takes a full-throttle action in each respective direction. Carlo The Carlo environment (seen in Figure 8.2 Fifth)is a simplified selfdriving simulator that uses a bicycle model for the dynamics. The agent observes its position, velocity, and heading in radians for a total of 5 dimensions. The agent has control over its heading and throttle, for an action space of [−1, 1]2 . For this domain, we have chosen to use a circular track where the agent starts in the center of the road at an angle of {π(1+2i)/4}3i=0 and drive counterclockwise around in a circle without crashing. Methods and Baseline When the action space is discrete, we use Q-learning with LCER otherwise we use PPO with LCER. The baseline we compare against is the same method without LCER. This allows us to verify the extent to which LCER impacts performance. We also plot a trajectory from the final policy for each environment in the middle of each column of Figure 8.2, except for Pacman as it is difficult to visualize the interaction between the ghost and pacman outside of video. Dealing with A. For PPO, the agent’s policy is a Gaussian (as in standard implementations) over the continuous action space. In order to deal with jump transitions (in the LDBA) when in a continuous action space (in the MDP), 95 we first let the agent decide whether to execute a jump transition or not (i.e. a probabilistic coin flip). If the agent chooses to not, then we take the action according to the Gaussian. The coin flip probability is learned, as well as the Gaussian. For the importance sampling term of PPO, the density of π is modified to account for the coin flip. For more details see Appendix F.1. Results Can we achieve desired behavior? The answer here is a resounding yes. For each environment (except Pacman) we illustrate the trajectory of the final policy above each learning curve in Figure 8.2. Determining the probability of satisfaction of the final policy is currently a challenging open problem (except in finite-state action spaces). Nevertheless, in each environment the agent qualitatively accomplishes the task. Even for challenging tasks with continuous action spaces, the agent is able to learn to accomplish the LTL specification. Does LCER help in the learning process? According to the learning curves in the last row of Figure 8.2, LCER demonstrably expedites learning. In every environment with the exception of Carlo, LCER generates significant lift over lack of experience replay. Intuition for why LCER helps? One way of viewing an LDBA is as a curriculum for what steps need to be taken in order to accomplish a task. By replacing the LDBA state of the agent with some other dream LDBA state, we are allowing the agent to “pretend” that it has already accomplished some portion of the task. As an example, consider the Flatworld example in Figure 8.2 with φ = GF (y & XF (r)) & (G¬b). A baseline agent (without LCER) would need to accomplish the entirety of the task in order to see any reward. However, an agent with counterfacual data, need only visit y from state 0 of the LDBA (see figure 8.1 for the LDBA). Then once the agent is really good at getting to y, it needs to learn how to reach r from state 2. After both of these tasks are accomplished, independently, the agent has solved the whole task. Placing the agent in state 0 of the LDBA effectively lets the agent pretend that it has already visited r. In this sense, part of the task has been accomplished. 96 8.5 Related Work Finding LTL-satisfying policies. Among the attempts at finding LTLsatisfying policies, Q-learning approaches have been the primary method of choice when the dynamics are unknown and Linear Programming methods when the dynamics are known [176, 83, 27, 33, 50]. The Q-learning approaches are predominantly constrained to finite state-action spaces. Among the works that extend to continuous action spaces [84], DDPG is used and takes the form of hierarchical RL which is known to potentially find myopic policies [205]. Handling a subset of LTL specifications involving those expressible as finite expressions can also be addressed with Reward machines [205, 34, 210]. Our work handles ω-regular expressions, subsuming regular expressions. Many problems are ω-regular problems, but not regular, such as liveness (something good will happen eventually) and safety (nothing bad will happen forever). On the formulation in Eq (8.3). Notable prior work on defining the value function as a function of the number of visits to S B and a state-dependent Γi function include Bozkurt et al. [27] and Cai et al. [32]. Most notably, these authors use multiple different state-dependent discount rates that have a complicated relationships between them that needs to be satisfied in the limit as γ → 1− . Our work drastically simplifies this, getting rid of the technical assumptions, while strengthening the guarantees. This allows us to find a non-asymptotic dependence on the suboptimality of a policies’ probability of LTL satisfaction as a function of γ. Off-policy data. One may view the counterfactual samples in Toro Icarte et al. [205] as an instantiation of LCER, limited to finite LTL expressions and discrete action spaces. Extension to continuous action space and full LTL requires a careful treatment. In the continuous action and full LTL setting, [218] incorporate starting the agent from a different initial LDBA state (than b−1 ) which is still on-policy but from a different starting state and doesn’t take advantage of the entire LDBA structure. This work can be seen as complimentary to our own. Theory. Works with strong theoretical guarantees on policy satisfaction include Fu and Topcu [67], Wolff et al. [221], and Voloshin et al. [215] but are once again limited to discrete state/action spaces. Extensions of these work to continuous state space are not trivial as they make heavy use of the discrete Markov chain structure afforded to them. 97 8.6 Discussion Our work, to the best of our knowledge, is the first to make full use of the LDBA as a form of experience replay and first to use policy gradient to learn LTL-satisfying policies. Our eventual discounting formulation is unrestricted to Finitary fragments of LTL like most prior work. Despite the guarantees afforded to us by eventual discounting, in general the problem given in Eq (8.2) is not PAC learnable [224]. Though, like SAT solvers, it is still useful to find reasonable heuristics to problems that are difficult. We show that under particular circumstances, eventual discounting gives a signal on the quantity of interest in (8.2) and even when it fails, it selects a policy that is difficult to differentiate from a successful one. Further, the bad news discussed in Section 8.2 we speculate is unavoidable in general LTL specifications, without significant assumptions on the MDP. For example, for stability problems in LTL and assuming control-affine dynamics then Lyapunov functions can serve as certificates for a policies’ LTL satisfaction. A reasonable relaxation to this would be require a system to behave a certain way for a long, but finite amount of time. 98 BIBLIOGRAPHY [1] Romina Abachi, Mohammad Ghavamzadeh, and Amir-massoud Farahmand. Policy-Aware Model Learning for Policy Gradient Methods. 2020. arXiv: 2003.00030 [cs.AI]. [2] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. “On the Expressivity of Markov Reward”. In: Advances in Neural Information Processing Systems. 2021. url: https://proceedings.neurips.cc/paper/ 2021/file/4079016d940210b4ae9ae7d41c4a2065-Paper.pdf. [3] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. “Constrained Policy Optimization”. In: International Conference on Machine Learning. 2017, pp. 22–31. [4] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. “Constrained Policy Optimization”. In: Proceedings of the 34th International Conference on Machine Learning. 2017. url: https : / / proceedings.mlr.press/v70/achiam17a.html (visited on 04/08/2022). [5] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudıók, John Langford, and Hanna Wallach. “A reductions approach to fair classification”. In: arXiv preprint arXiv:1803.02453 (2018). [6] Alekh Agarwal, Sham Kakade, and Lin F Yang. “Model-based reinforcement learning with a generative model is minimax optimal”. In: Conference on Learning Theory. 2020. [7] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. “Optimality and approximation with policy gradient methods in markov decision processes”. In: Conference on Learning Theory. 2020. [8] Gul Agha and Karl Palmskog. “A Survey of Statistical Model Checking”. In: ACM Trans. Model. Comput. Simul. 28.1 (Jan. 2018). issn: 1049-3301. doi: 10.1145/3158668. url: https://doi.org/10. 1145/3158668. [9] Eitan Altman. Constrained Markov decision processes. Vol. 7. CRC Press, 1999. [10] Eitan Altman. Constrained Markov Decision Processes: Stochastic Modeling. en. 1st ed. Boca Raton: Routledge, Dec. 2021. isbn: 978-1-315-14022-3. doi: 10 . 1201 / 9781315140223. url: https : / / www . taylorfrancis . com / books / 9781315140223 (visited on 04/08/2022). 99 [11] András Antos, Csaba Szepesvári, and Rémi Munos. “Fitted Qiteration in continuous action-space MDPs”. In: Advances in neural information processing systems. 2008, pp. 9–16. [12] András Antos, Csaba Szepesvári, and Rémi Munos. “Learning nearoptimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path”. In: Machine Learning 71.1 (2008), pp. 89–129. [13] Pranav Ashok, Jan Křetínský, and Maximilian Weininger. “PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games”. In: Computer Aided Verification. Ed. by Isil Dillig and Serdar Tasiran. Cham: Springer International Publishing, 2019, pp. 497– 519. isbn: 978-3-030-25540-4. [14] Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. “Model-based reinforcement learning with value-targeted regression”. In: International Conference on Machine Learning. 2020. [15] Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar. “Reinforcement learning of POMDPs using spectral methods”. In: Conference on Learning Theory. 2016. [16] Christel Baier and Joost-Pieter Katoen. Principles of model checking. en. Cambridge, Mass: The MIT Press, 2008. isbn: 978-0-262-02649-9. [17] Heejung Bang and James M. Robins. “Doubly Robust Estimation in Missing Data and Causal Inference Models”. In: Biometrics 61.4 (2005), pp. 962–973. doi: 10.1111/j.1541-0420.2005.00377.x. [18] Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. “Nearly-tight vc-dimension bounds for piecewise linear neural networks”. In: Proceedings of the 22nd Annual Conference on Learning Theory (COLT 2017). Ed. by Satyen Kale and Ohad Shamir. Vol. 65. Proceedings of Machine Learning Research. PMLR, July 2017, pp. 1064–1068. url: https://proceedings.mlr.press/v65/ harvey17a.html. [19] Peter L. Bartlett and Shahar Mendelson. “Rademacher and Gaussian Complexities: Risk Bounds and Structural Results”. In: Proceedings of the 14th Annual Conference on Computational Learning Theory and and 5th European Conference on Computational Learning Theory. COLT ’01/EuroCOLT ’01. Berlin, Heidelberg: Springer-Verlag, 2001, pp. 224–240. isbn: 3540423435. [20] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. “The arcade learning environment: An evaluation platform for general agents”. In: Journal of Artificial Intelligence Research 47 (2013), pp. 253–279. 100 [21] Dimitri P Bertsekas et al. “Dynamic programming and optimal control 3rd edition, volume ii”. In: Belmont, MA: Athena Scientific (2011). [22] Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control. Vol. 1. 3. Athena scientific Belmont, MA, 2005. [23] Lars Blackmore, Masahiro Ono, and Brian C Williams. “Chanceconstrained optimal path planning with obstacles”. In: IEEE Transactions on Robotics 27.6 (2011), pp. 1080–1094. [24] Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. “Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising”. In: Journal of Machine Learning Research (JMLR) 14 (2013), pp. 3207–3260. [25] Philippe Bougerol and Nico Picard. “Strict stationarity of generalized autoregressive processes”. In: The Annals of Probability 20.4 (1992), pp. 1714–1730. doi: 10.1214/aop/1176989526. url: https://doi. org/10.1214/aop/1176989526. [26] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. [27] Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. “Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning”. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). 2020, pp. 10349–10355. doi: 10.1109/ICRA40945.2020.9196796. [28] Tomáš Brázdil, Krishnendu Chatterjee, Martin Chmelík, Vojtěch Forejt, Jan Křetínský, Marta Kwiatkowska, David Parker, and Mateusz Ujma. “Verification of Markov Decision Processes Using Learning Algorithms”. In: Automated Technology for Verification and Analysis. Ed. by Franck Cassez and Jean-François Raskin. Cham: Springer International Publishing, 2014, pp. 98–114. isbn: 978-3-319-11936-6. [29] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. 2016. eprint: arXiv:1606.01540. [30] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. “OpenAI Gym”. In: CoRR abs/1606.01540 (2016). arXiv: 1606.01540. [31] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. “Openai gym”. In: arXiv preprint arXiv:1606.01540 (2016). 101 [32] Mingyu Cai, Mohammadhosein Hasanbeig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. “Modular deep reinforcement learning for continuous motion planning with temporal logic”. In: IEEE Robotics and Automation Letters 6.4 (2021), pp. 7973–7980. [33] Mingyu Cai, Shaoping Xiao, Zhijun Li, and Zhen Kan. “Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraints”. In: IEEE Transactions on Automatic Control (2021), pp. 1– 1. doi: 10.1109/TAC.2021.3138704. [34] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. “LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning”. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, July 2019, pp. 6065–6073. doi: 10.24963/ijcai.2019/840. url: https://doi.org/10.24963/ ijcai.2019/840. [35] Rich Caruana, Nikos Karampatziakis, and Ainur Yessenalina. “An empirical evaluation of supervised learning in high dimensions”. In: Proceedings of the 25th international conference on Machine learning. 2008, pp. 96–103. [36] Rich Caruana and Alexandru Niculescu-Mizil. “An empirical comparison of supervised learning algorithms”. In: Proceedings of the 23rd international conference on Machine learning. 2006, pp. 161–168. [37] Olivier Chapelle and Lihong Li. “An empirical evaluation of thompson sampling”. In: Advances in neural information processing systems. 2011, pp. 2249–2257. [38] Jinglin Chen and Nan Jiang. “Information-Theoretic Considerations in Batch Reinforcement Learning”. In: Proceedings of the 36th International Conference on Machine Learning. 2019, pp. 1042–1051. [39] Jinglin Chen and Nan Jiang. “Information-Theoretic Considerations in Batch Reinforcement Learning”. In: Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. Long Beach, California, USA: PMLR, June 2019, pp. 1042–1051. [40] Ching-An Cheng, Xinyan Yan, Evangelos Theodorou, and Byron Boots. “Accelerating Imitation Learning with Predictive Models”. In: Conference on Artificial Intelligence and Statistics (AISTATS). 2019. [41] Grace E. Cho and Carl D. Meyer. “Comparison of perturbation bounds for the stationary distribution of a Markov chain”. In: Linear 102 Algebra and its Applications 335.1 (2001), pp. 137–150. issn: 00243795. doi: https://doi.org/10.1016/S0024-3795(01)00320-2. [42] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models”. In: Advances in Neural Information Processing Systems 31. Ed. by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett. Curran Associates, Inc., 2018, pp. 4754–4765. [43] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. “Model-Based Reinforcement Learning via Meta-Policy Optimization”. In: 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings. Vol. 87. Proceedings of Machine Learning Research. PMLR, 2018, pp. 617–629. [44] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. “Coindice: Off-policy confidence interval estimation”. In: arXiv preprint arXiv:2010.11652 (2020). [45] Christoph Dann and Emma Brunskill. “Sample complexity of episodic fixed-horizon reinforcement learning”. In: Advances in Neural Information Processing Systems. Vol. 28. 2015, pp. 2818–2826. [46] Christoph Dann, Gerhard Neumann, and Jan Peters. “Policy evaluation with temporal differences: A survey and comparison”. In: The Journal of Machine Learning Research 15.1 (2014), pp. 809–883. [47] Thomas Degris, Martha White, and Richard S Sutton. “Off-Policy Actor-Critic”. In: (2012). [48] Marc Peter Deisenroth and Carl Edward Rasmussen. “PILCO: A Model-Based and Data-Efficient Approach to Policy Search”. In: Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML’11. Bellevue, Washington, USA: Omnipress, 2011, pp. 465–472. isbn: 9781450306195. [49] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li FeiFei. “Imagenet: A large-scale hierarchical image database”. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009, pp. 248–255. [50] Xuchu Ding, Stephen L. Smith, Calin Belta, and Daniela Rus. “Optimal Control of Markov Decision Processes With Linear Temporal Logic Constraints”. en. In: IEEE Transactions on Automatic Control 59.5 (May 2014), pp. 1244–1257. issn: 0018-9286, 1558-2523. doi: 10.1109/TAC.2014.2298143. [51] Victor Dorobantu and Andrew Taylor. LyaPy. https://github. com/vdorobantu/lyapy. 2020. 103 [52] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. “CARLA: An open urban driving simulator”. In: Conference on robot learning. 2017. [53] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. “Benchmarking deep reinforcement learning for continuous control”. In: International Conference on Machine Learning (ICML). 2016. [54] Miroslav Dudıók, John Langford, and Lihong Li. “Doubly robust policy evaluation and learning”. In: International Conference on Machine Learning (ICML). 2011. [55] Damien Ernst, Pierre Geurts, and Louis Wehenkel. “Tree-Based Batch Mode Reinforcement Learning”. In: J. Mach. Learn. Res. 6 (Dec. 2005), pp. 503–556. issn: 1532-4435. [56] Damien Ernst, Pierre Geurts, and Louis Wehenkel. “Tree-based batch mode reinforcement learning”. In: Journal of Machine Learning Research 6.4 (2005), pp. 503–556. [57] Amir M Farahmand, Mohammad Ghavamzadeh, Shie Mannor, and Csaba Szepesvári. “Regularized policy iteration”. In: Advances in Neural Information Processing Systems. IEEE. 2009, pp. 441–448. [58] Amir-massoud Farahmand. “Iterative Value-Aware Model Learning”. In: Advances in Neural Information Processing Systems 31. Ed. by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett. Curran Associates, Inc., 2018, pp. 9072–9083. [59] Amir-Massoud Farahmand, Andre Barreto, and Daniel Nikovski. “Value-Aware Loss Function for Model-based Reinforcement Learning”. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Ed. by Aarti Singh and Jerry Zhu. Vol. 54. Proceedings of Machine Learning Research. Fort Lauderdale, FL, USA: PMLR, Apr. 2017, pp. 1486–1494. [60] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. “More Robust Doubly Robust Off-policy Evaluation”. In: International Conference on Machine Learning (ICML). 2018. [61] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. “More Robust Doubly Robust Off-policy Evaluation”. In: arXiv preprint arXiv:1802.03493 (2018). [62] Yihao Feng, Lihong Li, and Qiang Liu. “A kernel loss for solving the bellman equation”. In: Advances in Neural Information Processing Systems. 2019, pp. 15430–15441. 104 [63] Chelsea Finn, Sergey Levine, and Pieter Abbeel. “Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization”. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. ICML’16. New York, NY, USA: JMLR.org, 2016, pp. 49–58. [64] Yoav Freund and Robert E Schapire. “Adaptive Game Playing Using Multiplicative Weights”. In: Games and Economic Behavior 29 (1999), pp. 79–103. [65] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. Springer Series in Statistics. New York, NY, USA: Springer, 2001. [66] Ronan Fruit, Matteo Pirotta, and Alessandro Lazaric. “Improved analysis of ucrl2 with empirical bernstein inequality”. In: arXiv preprint arXiv:2007.05456 (2020). [67] Jie Fu and Ufuk Topcu. “Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints”. In: Robotics: Science and Systems X, University of California, Berkeley, USA, July 12-16, 2014. Ed. by Dieter Fox, Lydia E. Kavraki, and Hanna Kurniawati. 2014. doi: 10.15607/RSS.2014.X.039. url: http: //www.roboticsproceedings.org/rss10/p39.html. [68] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning. 2020. arXiv: 2004.07219 [cs.LG]. [69] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. “Benchmarks for Deep Off-Policy Evaluation”. In: International Conference on Learning Representations. 2021. url: https://openreview.net/forum?id=kWSeGEeHvF8. [70] Javier García and Fernando Fernández. “A comprehensive survey on safe reinforcement learning”. In: Journal of Machine Learning Research 16.1 (2015), pp. 1437–1480. [71] Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Zhengxing Chen, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye. “Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform”. In: arXiv preprint arXiv:1811.00260 (2018). [72] Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. “Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model”. In: Machine learning 91.3 (2013), pp. 325–349. 105 [73] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. [74] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. “Generative Adversarial Nets”. In: Advances in Neural Information Processing Systems 27. Ed. by Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger. Curran Associates, Inc., 2014, pp. 2672–2680. [75] Zhaohan Guo, Philip S Thomas, and Emma Brunskill. “Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation”. In: Advances in Neural Information Processing Systems. 2017, pp. 2492–2501. [76] László Györfi, Michael Kohler, Adam Krzyzak, and Harro Walk. A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006. [77] David Ha and Jürgen Schmidhuber. “World Models”. In: arXiv preprint arXiv:1803.10122 (2018). [78] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. “Reinforcement Learning with Deep Energy-Based Policies”. In: International Conference on Machine Learning. 2017, pp. 1352–1361. [79] Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. “Cooperative inverse reinforcement learning”. In: Advances in Neural Information Processing Systems. 2016, pp. 3909–3917. [80] Ernst Moritz Hahn, Guangyuan Li, Sven Schewe, Andrea Turrini, and Lijun Zhang. “Lazy probabilistic model checking without determinisation”. In: arXiv preprint arXiv:1311.2928 (2013). [81] John Michael Hammersley and David Christopher Handscomb. “Monte Carlo methods”. In: (1964). [82] Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. “Q(lambda) with Off-Policy Corrections”. In: Conference on Algorithmic Learning Theory (ALT). 2016. [83] Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically-Constrained Reinforcement Learning. 2018. doi: 10. 48550/ARXIV.1801.08099. url: https://arxiv.org/abs/1801. 08099. [84] Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. “Deep reinforcement learning with temporal logics”. In: International Conference on Formal Modeling and Analysis of Timed Systems. Springer. 2020, pp. 1–22. 106 [85] Hado van Hasselt, Arthur Guez, and David Silver. “Deep Reinforcement Learning with Double Q-Learning”. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI’16. Phoenix, Arizona: AAAI Press, 2016, pp. 2094–2100. [86] Hado van Hasselt, Arthur Guez, and David Silver. “Deep Reinforcement Learning with Double Q-Learning”. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI’16. Phoenix, Arizona: AAAI Press, 2016, pp. 2094–2100. [87] David Haussler. “Sphere packing numbers for subsets of the Boolean n-cube with bounded Vapnik-Chervonenkis dimension”. In: Journal of Combinatorial Theory, Series A 69.2 (1995), pp. 217–232. [88] Mikael Henaff, Alfredo Canziani, and Yann LeCun. “Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic”. In: arXiv preprint arXiv:1901.02705 (2019). [89] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. “Deep reinforcement learning that matters”. In: Thirty-Second AAAI Conference on Artificial Intelligence. 2018. [90] Thomas Hérault, Richard Lassaigne, Frédéric Magniette, and Sylvain Peyronnet. “Approximate Probabilistic Model Checking”. In: Verification, Model Checking, and Abstract Interpretation. Ed. by Bernhard Steffen and Giorgio Levi. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 73–84. isbn: 978-3-540-24622-0. [91] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. “Deep q-learning from demonstrations”. In: ThirtySecond AAAI Conference on Artificial Intelligence. 2018. [92] Daniel G Horvitz and Donovan J Thompson. “A generalization of sampling without replacement from a finite universe”. In: Journal of the American statistical Association 47.260 (1952), pp. 663–685. [93] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. “Reward learning from human preferences and demonstrations in atari”. In: Advances in neural information processing systems 31 (2018). [94] Thomas Jaksch, Ronald Ortner, and Peter Auer. “Near-optimal Regret Bounds for Reinforcement Learning”. In: Journal of Machine Learning Research 11.51 (2010), pp. 1563–1600. issn: 1533-7928. [95] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. “When to Trust Your Model: Model-Based Policy Optimization”. In: Advances in Neural Information Processing Systems 32. Ed. by H. 107 Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett. Curran Associates, Inc., 2019, pp. 12519–12530. url: http : // papers . nips . cc/ paper / 9416 - when - to - trust - your model-model-based-policy-optimization.pdf. [96] Nan Jiang and Jiawei Huang. “Minimax Value Interval for OffPolicy Evaluation and Policy Optimization”. In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 2747–2758. [97] Nan Jiang and Lihong Li. “Doubly Robust Off-policy Value Evaluation for Reinforcement Learning”. In: International Conference on Machine Learning (ICML). 2016. [98] Nan Jiang and Lihong Li. “Doubly Robust Off-policy Value Evaluation for Reinforcement Learning”. In: International Conference on Machine Learning. 2016, pp. 652–661. [99] Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. “A composable specification language for reinforcement learning tasks”. In: Advances in Neural Information Processing Systems 32 (2019). [100] Sham Kakade and John Langford. “Approximately optimal approximate reinforcement learning”. In: ICML. Vol. 2. 2002, pp. 267–274. [101] Sham Machandranath Kakade. “On the sample complexity of reinforcement learning”. PhD thesis. University of College London, 2003. [102] Nathan Kallus and Masatoshi Uehara. “Double reinforcement learning for efficient off-policy evaluation in markov decision processes”. In: arXiv preprint arXiv:1908.08526 (2019). [103] Nathan Kallus and Masatoshi Uehara. “Efficiently breaking the curse of horizon: Double reinforcement learning in infinite-horizon processes”. In: arXiv preprint arXiv:1909.05850 (2019). [104] Joseph DY Kang, Joseph L Schafer, et al. “Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data”. In: Statistical science 22.4 (2007), pp. 523–539. [105] Abbas Kazerouni, Mohammad Ghavamzadeh, Yasin Abbasi Yadkori, and Benjamin Van Roy. “Conservative Contextual Linear Bandits”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. 2017. 108 [106] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL : Model-Based Offline Reinforcement Learning. 2020. arXiv: 2005.05951 [cs.LG]. [107] Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. url: http://arxiv.org/abs/1412.6980. [108] Jyrki Kivinen and Manfred K Warmuth. “Exponentiated Gradient versus Gradient Descent for Linear Predictors”. In: Information and Computation 132.1 (1997), pp. 1–63. [109] Sven Koenig and Reid G. Simmons. “The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms”. In: Machine Learning 22.1 (Mar. 1996), pp. 227–250. issn: 1573-0565. doi: 10.1007/BF00114729. [110] Andrey Kolobov, Mausam, and Daniel S. Weld. “A Theory of GoalOriented MDPs with Dead Ends”. In: Proceedings of the TwentyEighth Conference on Uncertainty in Artificial Intelligence. UAI’12. Catalina Island, CA: AUAI Press, 2012, pp. 438–447. [111] Jan Křetıónskỳ, Tobias Meggendorfer, and Salomon Sickert. “Owl: a library for ω-words, automata, and LTL”. In: International Symposium on Automated Technology for Verification and Analysis. Springer. 2018, pp. 543–550. [112] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. In: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1. NIPS’12. Lake Tahoe, Nevada: Curran Associates Inc., 2012, pp. 1097–1105. [113] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. “Model-Ensemble Trust-Region Policy Optimization”. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. [114] M. Kwiatkowska, G. Norman, and D. Parker. “PRISM 4.0: Verification of Probabilistic Real-time Systems”. In: Proc. 23rd International Conference on Computer Aided Verification (CAV’11). Ed. by G. Gopalakrishnan and S. Qadeer. Vol. 6806. LNCS. Springer, 2011, pp. 585–591. [115] Michail G Lagoudakis and Ronald Parr. “Least-squares policy iteration”. In: Journal of machine learning research 4.12 (2003), pp. 1107– 1149. 109 [116] Sascha Lange, Thomas Gabel, and Martin Riedmiller. “Batch reinforcement learning”. In: Reinforcement learning. Springer, 2012, pp. 45–73. [117] Richard Lassaigne and Sylvain Peyronnet. “Probabilistic Verification and Approximation”. In: Electron. Notes Theor. Comput. Sci. 143 (Jan. 2006), pp. 101–114. issn: 1571-0661. doi: 10.1016/j.entcs. 2005.05.031. url: https://doi.org/10.1016/j.entcs.2005.05. 031. [118] Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. “Finite-sample analysis of least-squares policy iteration”. In: Journal of Machine Learning Research 13.Oct (2012), pp. 3041–3074. [119] Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. “Finite-sample analysis of LSTD”. In: ICML-27th International Conference on Machine Learning. 2010, pp. 615–622. [120] Alessandro Lazaric and Marcello Restelli. “Transfer from multiple MDPs”. In: Advances in Neural Information Processing Systems. 2011, pp. 1746–1754. [121] Hoang M Le, Nan Jiang, Alekh Agarwal, Miro Dudík, Yisong Yue, and Hal Daumé III. “Hierarchical Imitation and Reinforcement Learning”. In: International Conference on Machine Learning (ICML). July 2018. [122] Hoang M Le, Andrew Kang, Yisong Yue, and Peter Carr. “Smooth imitation learning for online sequence prediction”. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48. JMLR. org. 2016, pp. 680–688. [123] Hoang M Le, Cameron Voloshin, and Yisong Yue. “Batch Policy Learning under Constraints”. In: International Conference on Machine Learning (ICML). 2019. [124] Wee Sun Lee, Peter L Bartlett, and Robert C Williamson. “Efficient agnostic learning of neural networks with bounded fan-in”. In: IEEE Transactions on Information Theory 42.6 (1996), pp. 2118–2132. [125] Sergey Levine and Pieter Abbeel. “Learning neural network policies with guided policy search under unknown dynamics”. In: Advances in Neural Information Processing Systems. 2014, pp. 1071–1079. [126] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. “Breaking the sample size barrier in model-based reinforcement learning with a generative model”. In: Advances in neural information processing systems 33 (2020), pp. 12861–12872. 110 [127] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. “Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms”. In: ACM Conference on Web Search and Data Mining (WSDM). 2011. [128] Lihong Li, Rémi Munos, and Csaba Szepesvári. “Toward Minimax Off-policy Value Estimation”. In: Artificial Intelligence and Statistics. 2015, pp. 608–616. [129] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. “Continuous control with deep reinforcement learning”. In: International Conference on Learning Representations (ICLR). 2016. [130] Long-Ji Lin. “Self-improving reactive agents based on reinforcement learning, planning and teaching”. In: Machine learning 8.3-4 (1992), pp. 293–321. [131] Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James MacGlashan. Environment-Independent Task Specifications via GLTL. 2017. doi: 10.48550/ARXIV.1704.04341. url: https: //arxiv.org/abs/1704.04341. [132] Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. “Robust regression for safe exploration in control”. In: Learning for Dynamics and Control (L4DC). 2020. [133] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. “Breaking the curse of horizon: Infinite-horizon off-policy estimation”. In: Neural Information Processing Systems (NeurIPS). 2018. [134] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. “Breaking the curse of horizon: Infinite-horizon off-policy estimation”. In: Advances in Neural Information Processing Systems. 2018, pp. 5361– 5371. [135] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. “Off-Policy Policy Gradient with State Distribution Correction”. In: arXiv preprint arXiv:1904.08473 (2019). [136] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations”. In: International Conference on Machine Learning. 2019, pp. 4114–4124. [137] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. “Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees”. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 111 [138] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H Chi. “Off-policy Learning in Twostage Recommender Systems”. In: International World Wide Web Conference (WWW). 2020. [139] David J. C. MacKay. Information Theory, Inference & Learning Algorithms. USA: Cambridge University Press, 2002. isbn: 0521642981. [140] Odalric-Ambrym Maillard, Rémi Munos, Alessandro Lazaric, and Mohammad Ghavamzadeh. “Finite-sample analysis of Bellman residual minimization”. In: Proceedings of 2nd Asian Conference on Machine Learning. 2010, pp. 299–314. [141] Andreas Maurer and Massimiliano Pontil. Empirical Bernstein Bounds and Sample Variance Penalization. 2009. doi: 10 . 48550 / ARXIV . 0907.3740. url: https://arxiv.org/abs/0907.3740. [142] Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire. “Reinforcement learning with convex constraints”. In: Neural Information Processing Systems (NeurIPS). 2019. [143] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. “Playing atari with deep reinforcement learning”. In: arXiv preprint arXiv:1312.5602 (2013). [144] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. “Human-level control through deep reinforcement learning”. In: Nature 518.7540 (2015), pp. 529–533. [145] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2012. [146] William H Montgomery and Sergey Levine. “Guided policy search via approximate mirror descent”. In: Advances in Neural Information Processing Systems. 2016, pp. 4008–4016. [147] Rémi Munos. “Error bounds for approximate policy iteration”. In: ICML. Vol. 3. 2003, pp. 560–567. [148] Rémi Munos. “Performance bounds in l_p-norm for approximate value iteration”. In: SIAM journal on control and optimization 46.2 (2007), pp. 541–561. [149] Remi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. “Safe and Efficient Off-Policy Reinforcement Learning”. In: Neural Information Processing Systems (NeurIPS). 2016. 112 [150] Rémi Munos and Csaba Szepesvári. “Finite-time bounds for fitted value iteration”. In: Journal of Machine Learning Research 9.5 (2008), pp. 815–857. [151] Susan A Murphy, Mark J van der Laan, James M Robins, and Conduct Problems Prevention Research Group. “Marginal mean models for dynamic regimes”. In: Journal of the American Statistical Association 96.456 (2001), pp. 1410–1423. [152] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. “Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections”. In: Advances in Neural Information Processing Systems. 2019, pp. 2315–2325. [153] Arkadii Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley, 1983. [154] Andrew Y Ng, Daishi Harada, and Stuart Russell. “Policy invariance under reward transformations: Theory and application to reward shaping”. In: Icml. Vol. 99. 1999, pp. 278–287. [155] Xinkun Nie, Emma Brunskill, and Stefan Wager. “Learning Whento-Treat Policies”. In: arXiv preprint arXiv:1905.09751 (2019). [156] Michael Oberst and David Sontag. “Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models”. In: International Conference on Machine Learning. 2019, pp. 4881–4890. [157] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. “SelfImitation Learning”. In: International Conference on Machine Learning. 2018. [158] Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, and J Balaram. “Chance-constrained dynamic programming with application to riskaware robotic space exploration”. In: Autonomous Robots 39.4 (2015), pp. 555–571. [159] Dirk Ormoneit and Śaunak Sen. “Kernel-based reinforcement learning”. In: Machine learning 49.2-3 (2002), pp. 161–178. [160] Ian Osband, Daniel Russo, and Benjamin Van Roy. “(More) efficient reinforcement learning via posterior sampling”. In: Advances in Neural Information Processing Systems 26 (2013). [161] Cosmin Paduraru. “Off-policy evaluation in Markov decision processes”. PhD thesis. McGill University Libraries, 2013. [162] Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. “Sample-efficient batch reinforcement learning for dialogue management optimization”. In: ACM Transactions on Speech and Language Processing (TSLP) 7.3 (2011), p. 7. 113 [163] Michael JD Powell and J Swann. “Weighted uniform sampling—a Monte Carlo technique for reducing variance”. In: IMA Journal of Applied Mathematics 2.3 (1966), pp. 228–236. [164] Doina Precup, Richard S Sutton, and Satinder P Singh. “Eligibility Traces for Off-Policy Policy Evaluation”. In: Proceedings of the 17th International Conference on Machine Learning. Morgan Kaufmann Publishers Inc. 2000, pp. 759–766. [165] Doina Precup, Richard S. Sutton, and Satinder P. Singh. “Eligibility Traces for Off-Policy Policy Evaluation”. In: International Conference on Machine Learning (ICML). 2000. [166] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. [167] Guannan Qu and Adam Wierman. “Finite-time analysis of asynchronous stochastic approximation and Q-learning”. In: Conference on Learning Theory. 2020. [168] Guannan Qu, Chenkai Yu, Steven Low, and Adam Wierman. “Exploiting Linear Models for Model-Free Nonlinear Control: A Provably Convergent Policy Gradient Approach”. In: 2021 60th IEEE Conference on Decision and Control (CDC). IEEE. 2021, pp. 6539–6546. [169] Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable Baselines3. https: //github.com/DLR-RM/stable-baselines3. 2019. [170] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A Game Theoretic Framework for Model Based Reinforcement Learning. 2020. arXiv: 2004.07804 [cs.LG]. [171] Jette Randlòv and Preben Alstròm. “Learning to Drive a Bicycle Using Reinforcement Learning and Shaping”. In: ICML. 1998. [172] Martin Riedmiller. “Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method”. In: Machine Learning: ECML 2005. Springer, 2005, pp. 317–328. [173] Martin Riedmiller, Thomas Gabel, Roland Hafner, and Sascha Lange. “Reinforcement learning for robot soccer”. In: Autonomous Robots 27.1 (2009), pp. 55–73. [174] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. “A survey of multi-objective sequential decision-making”. In: Journal of Artificial Intelligence Research 48 (2013), pp. 67–113. [175] Stephane Ross and J Andrew Bagnell. “Reinforcement and imitation learning via interactive no-regret learning”. In: arXiv preprint arXiv:1406.5979 (2014). 114 [176] Dorsa Sadigh, Eric S. Kim, Samuel Coogan, S. Shankar Sastry, and Sanjit A. Seshia. “A learning based approach to control synthesis of Markov decision processes for linear temporal logic specifications”. In: 53rd IEEE Conference on Decision and Control. 2014, pp. 1091–1096. doi: 10.1109/CDC.2014.7039527. [177] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. “A Large-scale Open Dataset for Bandit Algorithms”. In: arXiv preprint arXiv:2008.07146 (2020). [178] Florian Schaefer and Anima Anandkumar. “Competitive Gradient Descent”. In: Advances in Neural Information Processing Systems 32. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett. Curran Associates, Inc., 2019, pp. 7625– 7635. url: http://papers.nips.cc/paper/8979-competitivegradient-descent.pdf. [179] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. “Trust region policy optimization”. In: International Conference on Machine Learning. 2015, pp. 1889–1897. [180] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. “Proximal Policy Optimization Algorithms”. In: CoRR abs/1707.06347 (2017). arXiv: 1707.06347. url: http://arxiv. org/abs/1707.06347. [181] E. Seneta. “Perturbation of the stationary distribution measured by ergodicity coefficients”. In: Advances in Applied Probability 20.1 (1988), pp. 228–230. doi: 10.2307/1427277. [182] Shai Shalev-Shwartz et al. “Online learning and online convex optimization”. In: Foundations and Trends® in Machine Learning 4.2 (2012), pp. 107–194. [183] Jack Sherman and Winifred J. Morrison. “Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix”. In: Ann. Math. Statist. 21.1 (Mar. 1950), pp. 124–127. doi: 10.1214/aoms/1177729893. [184] Salomon Sickert, Javier Esparza, Stefan Jaax, and Jan Křetínský. “Limit-Deterministic Büchi Automata for Linear Temporal Logic”. In: Computer Aided Verification. Ed. by Swarat Chaudhuri and Azadeh Farzan. Cham: Springer International Publishing, 2016, pp. 312–332. isbn: 978-3-319-41540-6. [185] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. “Mastering the game of Go with deep neural networks and tree search”. In: Nature 529.7587 (2016), pp. 484–489. 115 [186] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. “Reward is enough”. In: Artificial Intelligence 299 (2021), p. 103535. [187] Jonathan Sorg. “The Optimal Reward Problem: Designing Effective Reward for Bounded Agents”. PhD thesis. University of Michigan, USA, 2011. url: https://hdl.handle.net/2027.42/89705. [188] Jonathan Sorg, Satinder Singh, and Richard L. Lewis. “Reward Design via Online Gradient Ascent”. In: Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 2. NIPS’10. Vancouver, British Columbia, Canada: Curran Associates Inc., 2010, pp. 2190–2198. [189] R.S. Sutton, D. Precup, and S. Singh. “Between MDPs and semiMDPs: A Framework for Temporal Abstraction in Reinforcement Learning”. In: Artificial Intelligence 112 (1999), pp. 181–211. [190] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, Mar. 1998. isbn: 0-26219398-1. [191] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. [192] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. [193] Richard S. Sutton. “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming”. In: In Proceedings of the Seventh International Conference on Machine Learning. Morgan Kaufmann, 1990, pp. 216–224. [194] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Second. The MIT Press, 2018. url: http : //incompleteideas.net/book/the-book-2nd.html. [195] Adith Swaminathan and Thorsten Joachims. “Batch learning from logged bandit feedback through counterfactual risk minimization.” In: Journal of Machine Learning Research 16.1 (2015), pp. 1731–1755. [196] Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. “Off-policy evaluation for slate recommendation”. In: Neural Information Processing Systems (NeurIPS). 2017. [197] Csaba Szepesvári. “Algorithms for reinforcement learning”. In: Synthesis lectures on artificial intelligence and machine learning 4.1 (2010), pp. 1–103. 116 [198] Jean Tarbouriech, Matteo Pirotta, Michal Valko, and Alessandro Lazaric. “Sample Complexity Bounds for Stochastic Shortest Path with a Generative Model”. en. In: Proceedings of the 32nd International Conference on Algorithmic Learning Theory. Mar. 2021. url: https://proceedings.mlr.press/v132/tarbouriech21a.html. [199] Jean Tarbouriech, Runlong Zhou, Simon Shaolei Du, Matteo Pirotta, Michal Valko, and Alessandro Lazaric. “Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret”. In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=cc_AXK6rWPJ. [200] Guy Tennenholtz, Shie Mannor, and Uri Shalit. “Off-Policy Evaluation in Partially Observable Environments”. In: arXiv preprint arXiv:1909.03739 (2019). [201] Philip Thomas and Emma Brunskill. “Data-efficient off-policy policy evaluation for reinforcement learning”. In: International Conference on Machine Learning (ICML). 2016. [202] Philip Thomas and Emma Brunskill. “Data-efficient off-policy policy evaluation for reinforcement learning”. In: International Conference on Machine Learning. 2016, pp. 2139–2148. [203] Philip S Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. “Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing”. In: AAAI Conference on Innovative Applications (IAA). 2017. [204] Emanuel Todorov, Tom Erez, and Yuval Tassa. “Mujoco: A physics engine for model-based control”. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE. 2012, pp. 5026– 5033. [205] Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. “Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning”. In: J. Artif. Int. Res. 73 (May 2022). issn: 1076-9757. doi: 10.1613/jair.1.12440. url: https://doi.org/10.1613/jair.1.12440. [206] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. “Is deep reinforcement learning really superhuman on atari? leveling the playing field”. In: arXiv preprint arXiv:1908.04683 (2019). [207] John Tsitsiklis and Benjamin Van Roy. “Analysis of temporal-diffference learning with function approximation”. In: Advances in neural information processing systems 9 (1996). 117 [208] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. “Minimax Weight and Q-Function Learning for Off-Policy Evaluation”. In: Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 1023–1032. [209] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. “Minimax weight and q-function learning for off-policy evaluation”. In: International Conference on Machine Learning. PMLR. 2020, pp. 9659–9668. [210] Pashootan Vaezipoor, Andrew C. Li, Rodrigo Toro Icarte, and Sheila A. McIlraith. “LTL2Action: Generalizing LTL Instructions for MultiTask RL”. In: Proceedings of the 38th International Conference on Machine Learning, ICML. Vol. 139. Proceedings of Machine Learning Research. 2021, pp. 10497–10508. url: http://proceedings.mlr. press/v139/vaezipoor21a.html. [211] Hado Van Hasselt, Arthur Guez, and David Silver. “Deep Reinforcement Learning with Double Q-Learning.” In: AAAI. Vol. 2. Phoenix, AZ. 2016, p. 5. [212] Kristof Van Moffaert and Ann Nowé. “Multi-objective reinforcement learning using sets of pareto dominating policies”. In: The Journal of Machine Learning Research 15.1 (2014), pp. 3483–3512. [213] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is All You Need”. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17. Long Beach, California, USA: Curran Associates Inc., 2017, pp. 6000–6010. isbn: 9781510860964. [214] Cameron Voloshin, Nan Jiang, and Yisong Yue. “Minimax Model Learning”. In: International Conference on Artificial Intelligence and Statistics. PMLR. 2021, pp. 1612–1620. [215] Cameron Voloshin, Hoang Minh Le, Swarat Chaudhuri, and Yisong Yue. “Policy Optimization with Linear Temporal Logic Constraints”. In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview.net/forum?id=yZcPRIZEwOG. [216] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. “Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning”. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Ed. by J. Vanschoren and S. Yeung. Vol. 1. 2021. url: https://openreview. net/forum?id=IsK8iKbL-I. 118 [217] John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior (commemorative edition). Princeton university press, 2007. [218] Chuanzheng Wang, Yinan Li, Stephen L. Smith, and Jun Liu. Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networks. 2020. doi: 10.48550/ARXIV.2004.02610. url: https://arxiv.org/abs/2004.02610. [219] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudík. “Optimal and Adaptive Off-policy Evaluation in Contextual Bandits”. In: International Conference on Machine Learning (ICML). 2017, pp. 3589– 3597. [220] MA Wiering. “Multi-agent reinforcement learning for traffic light control”. In: International Conference on Machine Learning (ICML). 2000. [221] Eric M. Wolff, Ufuk Topcu, and Richard M. Murray. “Robust control of uncertain Markov Decision Processes with temporal logic specifications”. In: 2012 IEEE 51st IEEE Conference on Decision and Control (CDC). 2012, pp. 3372–3379. doi: 10.1109/CDC.2012.6426174. [222] Yihong Wu and Pengkun Yang. “Chebyshev polynomials, moment matching, and optimal estimation of the unseen”. In: The Annals of Statistics 47.2 (2019), pp. 857–883. doi: 10.1214/17-AOS1665. url: https://doi.org/10.1214/17-AOS1665. [223] Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. “Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling”. In: Advances in Neural Information Processing Systems. 2019, pp. 9665–9675. [224] Cambridge Yang, Michael L. Littman, and Michael Carbin. “Reinforcement Learning for General LTL Objectives Is Intractable”. In: CoRR abs/2111.12679 (2021). arXiv: 2111.12679. url: https: //arxiv.org/abs/2111.12679. [225] Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. “Off-Policy Evaluation via the Regularized Lagrangian”. In: Advances in Neural Information Processing Systems 33 (2020). [226] Håkan L. S. Younes, Edmund M. Clarke, and Paolo Zuliani. “Statistical Verification of Probabilistic Properties with Unbounded Until”. In: Formal Methods: Foundations and Applications. Ed. by Jim Davies, Leila Silva, and Adenilso Simao. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 144–160. isbn: 978-3-642-19829-8. 119 [227] Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. “On the importance of hyperparameter optimization for model-based reinforcement learning”. In: International Conference on Artificial Intelligence and Statistics. 2021. [228] Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. “Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization”. In: arXiv preprint arXiv:2104.13877 (2021). [229] Shangtong Zhang, Bo Liu, and Shimon Whiteson. “GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values”. In: arXiv preprint arXiv:2001.11113 (2020). [230] Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. “The ai economist: Improving equality and productivity with ai-driven tax policies”. In: arXiv preprint arXiv:2004.13332 (2020). [231] Brian D Ziebart. “Modeling purposeful adaptive behavior with the principle of maximum causal entropy”. In: (2010). [232] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. “Maximum Entropy Inverse Reinforcement Learning”. In: AAAI. Vol. 8. Chicago, IL, USA. 2008, pp. 1433–1438. [233] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. “Maximum Entropy Inverse Reinforcement Learning”. In: Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3. AAAI’08. Chicago, Illinois: AAAI Press, 2008, pp. 1433– 1438. isbn: 9781577353683. [234] Martin Zinkevich. “Online convex programming and generalized infinitesimal gradient ascent”. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03). 2003, pp. 928–936. Part IV APPENDIX 121 Appendix A CHAPTER 1 APPENDIX A.1 Glossary of Terms Acronym Term OPE X A P R γ d0 D τ T N πb πe V, Q Off-Policy Policy Evaluation State Space Action Space Transition Function Reward Function Discount Factor Initial State Distribution Dataset Trajectory/Episode Horizon/Episode Length Number of episodes in D Behavior Policy Evaluation Policy Value (ex: V (s)), Action-Value (ex: Q(s, a)) ρij:j ′ Cumulative Importance Weight, Qmin(j ′ ,T −1) πe (ait |sit ) t=j πb (ait |sit ) . If j > j ′ then ρ = 1. IPS Inverse Propensity Scoring DM Direct Method HM Hybrid Method IS Importance Sampling PDIS Per-Decision Importance Sampling WIS Weighted Importance Sampling PDWIS Per-Decision Weighted Importance Sampling FQE Fitted Q Evaluation [123] IH Infinite Horizon [133] Q-Reg Q Regression [60] MRDR More Robust Doubly Robst [60] AM Approximate Model (Model Based) Q(λ) Qπ (λ) [82] R(λ) Retrace(λ) [149] Tree Tree-Backup(λ) [165] DR Doubly-Robust [97, 54] WDR Weighted Doubly-Robust [54] MAGIC Model And Guided Importance Sampling Combining (Estimator) [201] Graph Graph Environment Graph-MC Graph Mountain Car Environment MC Mountain Car Environment Pix-MC Pixel-Based Mountain Car Environment Enduro Enduro Environment Graph-POMDP Graph-POMDP Environment GW Gridworld Environment Pix-GW Pixel-Based Gridworld Environment 122 A.2 Ranking of Methods A method that is within 10% of the method with the lowest Relative MSE is counted as a top method, called Near-top Frequency, and then we aggregate across all experiments. See Table A.1 for a sorted list of how often the methods appear within 10% of the best method. Table A.1: Fraction of time among the top estimators across all experiments Method MAGIC FQE DM FQE IH WDR FQE MAGIC Qπ (λ) WDR Qπ (λ) DM Qπ (λ) DR Qπ (λ) WDR R(λ) DR FQE MAGIC R(λ) WDR Tree DR R(λ) DM R(λ) DM Tree MAGIC Tree DR Tree DR MRDR WDR Q-Reg DM AM IS WDR MRDR PDWIS DR Q-Reg MAGIC AM MAGIC MRDR DM MRDR PDIS MAGIC Q-Reg WIS NAIVE DM Q-Reg DR AM WDR AM Near-top Frequency 0.300211 0.236786 0.190275 0.177590 0.173362 0.173362 0.150106 0.135307 0.133192 0.128964 0.107822 0.105708 0.105708 0.097252 0.084567 0.076110 0.073996 0.073996 0.071882 0.065539 0.063425 0.054968 0.046512 0.044397 0.038055 0.033827 0.033827 0.033827 0.027484 0.025370 0.025370 0.019027 0.012685 0.006342 123 Decision Tree Support Tables A.2-A.9 provide a numerical support for the decision tree in Figure 3.3. Each table refers to a child node in the decision tree, ordered from left to right, respectively. For example, Table A.2 refers to the left-most child node (propery specified, short horizon, small policy mismatch) while Table A.9 refers to the right-most child node (misspecified, good representation, long horizon, good πb estimate). Table A.2: Near-top Frequency. Properly specified, short horizon, small policy mismatch experiments. Table A.3: Near-top Frequency. Properly specified, short horizon, large policy mismatch experiments. DM Hybrid DM Hybrid Direct DR WDR MAGIC Direct DR WDR MAGIC AM 4.7% 4.7% 3.1% Q-Reg 0.0% 4.7% 6.2% MRDR 7.8% 14.1% 7.8% FQE 40.6% 23.4% 21.9% R(λ) 17.2% 20.3% 20.3% Qπ (λ) 21.9% 18.8% 18.8% Tree 15.6% 12.5% 12.5% IH 17.2% - 4.7% 4.7% 7.8% 34.4% 14.1% 17.2% 14.1% - AM 20.3% 1.6% 0.0% Q-Reg 1.6% 1.6% 3.1% MRDR 3.1% 1.6% 6.2% FQE 35.9% 14.1% 17.2% R(λ) 23.4% 14.1% 20.3% Qπ (λ) 15.6% 15.6% 14.1% Tree 21.9% 12.5% 18.8% IH 29.7% - IPS IPS Standard Per-Decision 4.7% 3.1% 1.6% IS WIS NAIVE 4.7% 3.1% - Table A.4: Near-top Frequency. Poperly specified, long horizon, small policy mismatch experiments. DM Hybrid Direct DR WDR MAGIC Standard Per-Decision IS WIS NAIVE 0.0% 0.0% 3.1% Direct 5.6% 1.4% 2.8% 50.0% 9.7% 18.1% 2.8% - 0.0% 1.6% - Table A.5: Near-top Frequency. Properly specified, long horizon, large policy mismatch, deterministic env/rew experiments. DM AM 6.9% 0.0% 0.0% Q-Reg 0.0% 1.4% 1.4% MRDR 1.4% 0.0% 1.4% FQE 50.0% 22.2% 23.6% R(λ) 13.9% 12.5% 11.1% Qπ (λ) 20.8% 18.1% 18.1% Tree 2.8% 1.4% 0.0% IH 29.2% - 7.8% 1.6% 1.6% 37.5% 23.4% 20.3% 21.9% - Hybrid DR WDR MAGIC AM 3.5% 3.5% 1.8% Q-Reg 3.5% 1.8% 0.0% MRDR 3.5% 1.8% 0.0% FQE 15.8% 17.5% 29.8% R(λ) 1.8% 3.5% 0.0% Qπ (λ) 22.8% 15.8% 38.6% Tree 3.5% 3.5% 1.8% IH 21.1% - 1.8% 0.0% 0.0% 28.1% 0.0% 24.6% 1.8% - IPS IPS Standard Per-Decision Standard Per-Decision IS WIS NAIVE 0.0% 0.0% 5.6% 0.0% 0.0% - IS WIS NAIVE 5.3% 0.0% 0.0% 3.5% 8.8% - 124 Table A.6: Near-top Frequency. Properly specified, long horizon, large policy mismatch, stochastic env/rew experiments. Table A.7: Near-top Frequency. Potentially misspecified, insufficient representation experiments. DM DM Hybrid Direct DR WDR MAGIC Direct AM 14.6% Q-Reg 4.2% MRDR 4.2% FQE 31.2% R(λ) 4.2% Qπ (λ) 2.1% Tree 4.2% IH 41.7% 0.0% 2.1% 2.1% 2.1% 6.2% 0.0% 6.2% - 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% - 8.3% 2.1% 0.0% 25.0% 0.0% 2.1% 0.0% - AM Q-Reg MRDR FQE R(λ) Qπ (λ) Tree IH Hybrid DR WDR MAGIC 3.9% 13.7% 25.5% 0.0% 18.6% 15.7% 0.0% 5.9% 13.7% 6.9% - 6.9% 5.9% 24.5% - IPS IPS Standard Per-Decision Standard Per-Decision 25.0% 0.0% 2.1% IS WIS NAIVE 4.2% 0.0% - Table A.8: Near-top Frequency. Potentially misspecified, sufficient representation, poor πb estimate experiments. IS WIS NAIVE 10.8% 9.8% 3.9% 8.8% 13.7% - Table A.9: Near-top Frequency. Potentially misspecified, sufficient representation, good πb estimate experiments. DM Hybrid DM Direct DR WDR MAGIC Direct DR 0.0% 0.0% 0.0% 8.3% 2.8% 5.6% 5.6% 0.0% 0.0% 0.0% 0.0% 0.0% 5.6% 0.0% 8.3% 25.0% 8.3% 8.3% 5.6% 8.3% 8.3% 16.7% - AM 0.0% Q-Reg 0.0% MRDR 13.3% FQE 0.0% R(λ) 16.7% Qπ (λ) 6.7% Tree 20.0% IH 0.0% IS WIS NAIVE 0.0% 0.0% 6.7% 3.3% 0.0% 0.0% 0.0% - 0.0% 3.3% 0.0% 6.7% 6.7% 0.0% 6.7% - 0.0% 0.0% 0.0% 10.0% 20.0% 3.3% 6.7% - AM Q-Reg MRDR FQE R(λ) Qπ (λ) Tree IH Hybrid WDR MAGIC 2.8% 0.0% 5.6% 11.1% 19.4% 0.0% 5.6% - IPS IPS Standard Per-Decision Standard Per-Decision 3.3% 0.0% 0.0% 0.0% 0.0% - IS WIS NAIVE 0.0% 0.0% 0.0% 0.0% 0.0% - 125 A.3 Supplementary Folklore Backup The following tables represent the numerical support for how horizon and policy difference affect the performance of the OPE estimators when policy mismatch is held constant. Notice that the policy mismatch for table A.11 and  100  10 .9 A.12 are identical: .124573... = . What we see here is that despite .1 .1 identical policy mismatch, the longer horizon does not impact the error as much (compared to the baseline, Table A.10) as moving πe to .9, far from .1 and keeping the horizon the same. For the tables in this section we have T = 10, N = 50, πb (a = 0) = 0.1, πe (a = 0) = 0.1246, unless otherwise specified. Table A.10: Graph, relative MSE. Dense rewards. Baseline. DM Table A.11: Graph, relative MSE. Dense rewards. T = 100. Increasing horizon compared to baseline, fixed πe . Hybrid DM Direct DR AM 1.9E-3 Q-Reg 2.4E-3 MRDR 5.8E-3 FQE 1.8E-3 R(λ) 1.8E-3 Qπ (λ) 1.8E-3 Tree 1.8E-3 IH 1.6E-3 4.9E-3 4.3E-3 8.9E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - Hybrid WDR MAGIC Direct 5.0E-3 4.2E-3 9.4E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - 3.4E-3 4.5E-3 9.2E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - AM 5.6E-2 Q-Reg 3.4E-3 MRDR 1.1E-2 FQE 6.0E-2 R(λ) 6.0E-2 Qπ (λ) 6.0E-2 Tree 3.4E-1 IH 4.7E-4 DR WDR MAGIC 5.9E-2 5.9E-2 1.1E-1 1.2E-1 2.5E-1 2.9E-1 6.0E-2 6.0E-2 6.0E-2 6.0E-2 6.0E-2 6.0E-2 7.0E-3 1.6E-3 - 5.3E-2 9.2E-2 3.1E-1 6.0E-2 6.0E-2 6.0E-2 2.3E-3 - IPS IPS Standard Per-Decision Standard Per-Decision IS WIS NAIVE 5.6E-4 1.4E-3 6.1E-3 8.4E-4 1.4E-3 - IS WIS NAIVE 1.7E-2 9.5E-4 5.4E-3 2.5E-3 4.9E-4 - Table A.12: Graph, relative MSE. πe (a = 0) = 0.9. Dense rewards. Increasing πe compared to baseline, fixed horizon. DM Direct Hybrid DR WDR MAGIC AM 6.6E-1 6.7E-1 6.6E-1 Q-Reg 5.4E-1 6.3E-1 1.3E0 MRDR 5.4E-1 7.3E-1 2.0E0 FQE 6.6E-1 6.6E-1 6.6E-1 R(λ) 6.7E-1 6.6E-1 9.3E-1 Qπ (λ) 6.6E-1 6.6E-1 6.6E-1 Tree 6.7E-1 6.6E-1 9.4E-1 IH 1.4E-2 - 6.6E-1 9.3E-1 2.0E0 6.6E-1 1.0E0 6.6E-1 1.0E0 - IPS Standard Per-Decision IS WIS NAIVE 1.0E0 2.0E0 4.0E0 5.4E-1 9.7E-1 - 126 A.4 Model Selection Guidelines For the definition of near-top frequency, see the definition in Section 3.1. For support of the guideline, see Table A.1) Table A.13: Model Selection Guidelines. Class Direct Recommendation When to use FQE Q(λ) IH IPS PDWIS Hybrid MAGIC FQE MAGIC Q(λ) Prototypical env. Stochastic env, severe policy mismatch Compute non-issue, moderate policy mismatch Long horizon, mild policy mismatch, good kernel Short horizon, mild policy mismatch Severe model misspecification Compute non-issue, severe model misspecification Graph, MC, Pix-MC GW/Pix-GW Graph-MC Graph Graph-POMDP, Enduro Graph-POMDP Table A.14: Model Selection Guideline Cont. Class Direct Recommendation Near-top Freq. FQE Q(λ) IH IPS PDWIS Hybrid MAGIC FQE MAGIC Q(λ) 23.7% 15.0% 19.0% 4.7% 30.0% 17.3% 127 A.5 Methods Below we include a description of each of the methods we tested. Let T̃ = T − 1. Inverse Propensity Scoring (IPS) Methods Table A.15: IPS methods. [54, 97] Standard Per-Decision ρi PT̃ t 0:T̃ t=0 γ rt i=1 N PN ρi0:T̃ PT̃ t i=1 w0:T̃ t=0 γ rt t ρ0:t r N t PN PT̃ ρi0:t t i=1 t=0 γ w0:t rt PN IS WIS PN PT̃ i=1 i t=0 γ Table A.15 shows the calculation for the four traditional IPS estimators: VIS , VP DIS , VW IS , VP DW IS . In addition, we include the following method as well since it is a Rao-Blackwellization [133] of the IPS estimators: Hybrid Methods b an estimate of Q, Hybrid rely on being supplied an action-value function Q, P b from which one can also yield Vb (s) = a∈A π(a|s)Q(s, a). Doubly-Robust (DR): [201, 97] ∞ N N X 1 X 1 X b i , ai ) + γ Vb (si )]. γ t ρi0:t [rti − Q(s Vb (si0 ) + t t t+1 N i=1 N i=1 t=0 Weighted Doubly-Robust (WDR): [201] VDR = VW DR = N N X ∞ X 1 X ρi b i , ai ) + γ Vb (si )]. γ t 0:t [rti − Q(s Vb (si0 ) + t t t+1 N i=1 w 0:t i=1 t=0 MAGIC: [201] Given gJ = {g i |i ∈ J ⊆ N ∪ {−1}} where j N N X X ρi0:t i X ρi ρi b i i ρi0:T̃ b i rt + γ j+1 0:t Vb (sij+1 )− V (st )), γ t ( 0:t Q(s t , at )− w0:t w0:t w0:t w0:T̃ i=1 t=0 i=1 i=1 t=0 then define dist(y, Z) = minz∈Z |y − z| and g j (D) = j N X X γt b b (j) = dist(g J (D), CI(g ∞ (D), 0.5)) n j b (i, j) = Cov(g J (D), g J (D)), Ω n i then for a |J|−simplex ∆ |J| j we can calculate b +b xb∗ ∈ arg min xT [Ω bbbT ]x, n x∈∆|J| yielding VM AGIC = (xb∗ )T gJ . MAGIC can be thought of as a weighted average of different blends of the DM 128 and Hybrid. In particular, for some i ∈ J, g i represents estimating the first i steps of V (πe ) according to DR (or WDR) and then estimating the remaining b Hence, V steps via Q. M AGIC finds the most appropriate set of weights which trades off between using a direct method and a Hybrid. Direct Methods (DM) Model-Based Approximate Model (AM): [97] An approach to model-based value estimation is to directly fit the transition dynamics P (st+1 |st , at ), reward R(st , at ), and terminal condition P (st+1 ∈ Sterminal |st , at ) of the MDP using some for of maximum likelihood or function approximation. This yields a simulation environment from which one can extract the value of a policy using an average P over rollouts. Thus, V (π) = E[ Tt=1 γ t r(st , at )|s0 = s, a0 = π(s0 )] where the expectation is over initial conditions s ∼ d0 and the transition dynamics of the simulator. Model-Free b θ), parametrized Every estimator in this section will approximate Q with Q(·; b the OPE estimate we seek is by some θ. From Q N X 1 X b i , a; θ). πe (a|s)Q(s V = 0 N i=1 a∈A Note that Eπe Q(st+1 , ·) = a∈A πe (a|st+1 )Q(st+1 , a). P Direct Model Regression (Q-Reg): [60] b θ) = min Q(·, θ Rt:i T̃ = T̃ X N X T̃   1 X b i , ai ; θ) 2 γ t ρi0:t Rt:i T̃ − Q(s t t N i=1 t=0 ′ γ t −t ρi(t+1):t′ rti′ t′ =t b θ) = lim b Fitted Q Evaluation (FQE): [123] Q(·, k→∞ Qk where b = min Q k θ N X T̃ 1 X i i i 2 b (Q k−1 (st , at ; θ) − yt ) N i=1 t=0 i b yti ≡ rti + γEπe Q k−1 (st+1 , ·; θ) Retrace(λ) (R(λ)), Tree-Backup (Tree), Qπ (λ): [149, 165, 82] b θ) = lim Q b Q(·, k k→∞ 129 where b (s, a; θ) = Q b Q k k−1 (s, a; θ) + Eπb [ X γt t≥0 t Y ck yt |s0 = s, a0 = a] k=1 and b b yt = rt + γEπe Q k−1 (st+1 , ·; θ) − Qk−1 (st , at ; θ) ck =  (ak |sk )   ) λ min(1, ππeb (a   k |sk )  λπe (ak |sk )     λ R(λ) T ree . Qπ (λ) More Robust Doubly-Robust (MRDR): [60] Given Ωπb (s) = diag[1/πb (a|s)]a∈A − eeT e = [1, . . . , 1]T Rt:i T̃ = T̃ X γ j−t ρi(t+1):j r(sij , aij ) j=t and b qθ (s, a, r) = diag[πe (a′ |s)]a′ ∈A [Q(s, a′ ; θ)]a′ ∈A − r[1{a′ = a}]a′ ∈A where 1 is the indicator function, then b θ) = min Q(·, θ N X T̃ 1 X γ 2t (ρi0:T̃ )2 × ρit qθ (sit , ait , Rt:i T̃ )T Ωπb (sit )qθ (sit , ait , Rt:i T̃ ). N i=1 t=0 State Density Ratio Estimation (IH): [133] VIH = γ t ω(sit )ρt:t rti T̃ N X X PT̃ i′ t′ i′ =0 t′ =1 γ ω(st′ )ρt′ :t′ PT t i t=0 γ dπe (st ) i , ω(st ) = lim PT i t t→∞ t=0 γ dπb (st ) i=1 t=0 PN where πb is assumed to be a fixed data-generating policy, and dπ is the distribution of states when executing π from s0 ∼ d0 . The details for how to find ω can be found in Algorithm 1 and 2 of [133]. 130 A.6 Environments For every environment, we initialize the environment with a fixed horizon length T . If the agent reaches a goal before T or if the episode is not over by step T , it will transition to an environment-dependent absorbing state where it will stay until time T . For a high level description of the environment features, see Table 3.1. R = +1 or R ~ ((+1,1) a=0 a=0 1 a=1 3 a= 2T-3 a=1 -10 0 10 11 0 + R= ABS 1 ABS a=0 a=1 2 a=1 4 2T-2 a= 0 -1 a= 0 1 Figure A.3: MC Environment, pixel-version. The Figure A.1: Graph Envi- Figure A.2: Graph-MC non-pixel version involves ronment. Environment. representing the state of the car as the position and velocity. R = −1 or R ~ ((−1,1) a= 1 a= 1 0 S S S o=1 o=0 a=0 a=0 1 o=T-1 S F S S 3 S S S S S H H F F H F 2T-3 S H F a=1 0 ABS a=0 a=1 2 a=1 4 2T-2 S H S H S H F H H H H F G Figure A.6: Gridworld Figure A.5: Graph- environment. Blank POMDP Environment. spaces indicate areas of Model-Fail [201] is a a small negative reward, special case of this S indicates the starting environment where T=2. states, F indicates a field We also extend the of slightly less negative reenvironment to arbitrary ward, H indicates a hole Figure A.4: Enduro En- horizon which makes it a of severe penalty, G indivironment. semi-mdp. cates the goal of positive reward. Environment Descriptions Graph Figure A.1 shows a visualization of the Toy-Graph environment. The graph is initialized with horizon T and with absorbing state sabs = 2T . In each episode, 131 the agent starts at a single starting state s0 = 0 and has two actions, a = 0 and a = 1. At each time step t < T , the agent can enter state st+1 = 2t + 1 by taking action a = 0, or st+1 = 2t + 2 by taking action a = 1. If the environment is stochastic, we simulate noisy transitions by allowing the agent to slip into st+1 = 2t + 2 instead of st+1 = 2t + 1 and vice-versa with probability .25. At the final time t = T , the agent always enters the terminal state sabs . The reward is +1 if the agent transitions to an odd state, otherwise is −1. If the environment provides sparse rewards, then r = +1 if sT −1 is odd, r = −1 if sT −1 is even, otherwise r = 0. Similarly to deterministic rewards, if the environment’s rewards are stochastic, then the reward is r ∼ N (1, 1) if the agent transitions to an odd state, otherwise r ∼ N (−1, 1). If the rewards are sparse and stochastic then r ∼ N (1, 1) if sT −1 is odd, otherwise r ∼ N (−1, 1) and r = 0 otherwise. Graph-POMDP Figure A.5 shows a visualization of the Graph-POMDP environment. The underlying state structure of Graph-POMDP is exactly the Graph environment. However, the states are grouped together based on a choice of Graph-POMDP horizon length, H. This parameter groups states into H observable states. The agent only is able to observe among these states, and not the underlying MDP structure. Model-Fail [201] is a special case of this environment when H = T = 2. Graph Mountain Car (Graph-MC) Figure A.2 shows a visualization of the Toy-MC environment. This environment is a 1-D graph-based simplification of Mountain Car. The agent starts at s0 = 0, the center of the valley and can go left or right. There are 21 total states, 10 to the left of the starting position and 11 to the right of the starting position, and a terminal absorbing state sabs = 22. The agent receives a reward of r = −1 at every timestep. The reward becomes zero if the agent reaches the goal, which is state x = +11. If the agent reaches x = −10 and continues left then the agent remains in x = −10. If the agent does not reach state x = +11 by step T then the episode terminates and the agent transitions to the absorbing state. 132 Mountain Car (MC) We use the OpenAI version of Mountain Car with a few simplifying modifications [29, 192]. The car starts in a valley and has to go back and forth to gain enough momentum to scale the mountain and reach the end goal. The state space is given by the position and velocity of the car. At each time step, the car has the following options: accelerate backwards, forwards or do nothing. The reward is r = −1 for every time step until the car reaches the goal. While the original trajectory length is capped at 200, we decrease the effective length by applying every action at five times before observing st+1 . Furthermore, we modify the random initial position from being uniformly between [−.6, −.4] to being one of {−.6, −.5, −.4}, with no velocity. The environment is initialized with a horizon T and absorbing state sabs = [.5, 0], position at .5 and no velocity. Pixel-based Mountain Car (Pix-MC) This environment is identical to Mountain Car except the state space has been modified from position and velocity to a pixel based representation of a ball, representing a car, rolling on a hill, see Figure A.3. Each frame ft is a 80 × 120 image of the ball on the mountain. One cannot deduce velocity from a single frame, so we represent the state as st = {ft−1 , ft } where f−1 = f0 , the initial state. Everything else is identical between the pixel-based version and the position-velocity version described earlier. Enduro We use OpenAI’s implementation of Enduro-v0, an Atari 2600 racing game. We downsample the image to a grayscale of size (84,84). We apply every action one time and we represent the state as st = {ft−3 , ft−2 , ft−1 , ft } where fi = f0 , the initial state, for i < 0. See Figure A.4 for a visualization. Gridworld (GW) Figure A.6 shows a visualization of the Gridworld environment. The agent starts at a state in the first row or column (denoted S in the figure), and proceeds through the grid by taking actions, given by the four cardinal directions, for T = 25 timesteps. An agent remains in the same state if it chooses an action which would take it out of the environment. If the agent reaches the goal state Table A.16: Environment parameters - Full description. 133 Environment Graph Graph-MC MC Pix-MC Enduro Graph-POMDP GW Pix-GW Is MDP? State desc. T Stoch Env? Stoch Rew? Sparse Rew? Q̂ Func. Class Initial state Absorb. state Frame height Frame skip yes position 4 or 16 variable variable variable tabular 0 2T 1 1 yes position 250 no no terminal tabular 0 22 1 1 yes pixels 250 no no terminal NN variable img([.5,0]) 2 5 yes pixels 1000 no no dense NN gray img zero img 4 1 no position 2 or 8 no no terminal tabular 0 2T 1 1 yes pixels 25 variable no dense NN variable zero img 1 1 yes [pos, vel] 250 no no terminal linear/NN variable [.5,0] 2 5 yes position 25 no no dense tabular variable 64 1 1 G, in the bottom right corner of the environment, it transitions to a terminal state x = 64 for the remainder of the trajectory and receives a reward of +1. In the grid, there is a field (denoted F) which gives the agent a reward of −.005 and holes (denoted H) which give −.5. The remaining states give a reward of −.01. Pixel-Gridworld (Pixel-GW) This environment is identical to Gridworld except the state space has been modified from position to a pixel based representation of the position: 1 for the agent’s location, 0 otherwise. We use the same policies as in the Gridworld case. A.7 Experimental Setup Description of the policies Graph, Graph-POMDP and Graph-MC use static policies with some probability of going left and another probability of going right, ex: π(a = 0) = p, π(a = 1) = 1 − p, independent of state. We vary p in our experiments. GW, Pix-GW, MC, Pixel-MC, and Enduro all use an ϵ−Greedy policy. In other words, we train a policy Q∗ (using value iteration or DDQN) and then vary the deviation away from the policy. Hence ϵ − Greedy(Q∗ ) implies we follow a mixed policy π = arg maxa Q∗ (s, a) with probability 1 − ϵ and uniform with probability ϵ. We vary ϵ in our experiments. Enumeration of Experiments The set of experiments per table is the Cartesian product of the parameters in the table. 134 Table A.17: Graph parameters. Table A.18: Graph-POMDP parameters. Parameters Parameters γ .98 N 23:11 T {4, 16} πb (a = 0) {.2, .6} πe (a = 0) .8 Stochastic Env {True, False} Stochastic Rew {True, False} Sparse Rew {True, False} Seed {10 random} ModelType Tabular Regress πb False Table A.19: Gridworld parameters. γ .98 N 28:11 (T,H) {(2, 2), (16, 6)} πb (a = 0) {.2, .6} πe (a = 0) .8 Stochastic Env {True, False} Stochastic Rew {True, False} Sparse Rew {True, False} Seed {10 random} ModelType Tabular Regress πb False Table A.20: Pix-GW parameters. Parameters Parameters γ .98 N 26:11 T 25 ϵ − Greedy, πb {.2, .4, .6, .8, 1.} ϵ − Greedy, πe .1 Stochastic Env False Stochastic Rew False Sparse Rew False Seed {10 random} ModelType Tabular Regress πb True γ .96 N 26:9 T 25 ϵ − Greedy, πb {.2, .4, .6, .8, 1.} ϵ − Greedy, πe .1 Stochastic Env {True, False} Stochastic Rew False Sparse Rew False Seed {10 random} ModelType NN Regress πb {True, False} Table A.21: Graph-MC parameters. Table A.22: MC parameters. Parameters Parameters γ N T .99 27:11 250 {(.45, .45), (.6, .6), (.45.6) (πb (a = 0), πe (a = 0)) (.6, .45), (.8, .2), (.2, .8)} Stochastic Env False Stochastic Rew False Sparse Rew False Seed {10 random} ModelType Tabular Regress πb False γ N T .99 27:10 250 {(.1, 0), (1, 0) ϵ − Greedy, (πb , πe ) (1, .1), (.1, 1)} Stochastic Env False Stochastic Rew False Sparse Rew False Seed {10 random} ModelType {Tabular, NN} Regress πb False 135 Table A.23: Pix-MC parameters. Table A.24: Enduro parameters. Parameters γ N T .97 512 500 {(.25, 0), (.1, 0) ϵ − Greedy, (πb , πe ) (.25, .1)} Stochastic Env False Stochastic Rew False Sparse Rew False Seed {10 random} ModelType {Tabular, NN} Regress πb False Parameters γ N T .9999 512 500 {(.25, 0), (.1, 0) ϵ − Greedy, (πb , πe ) (.25, .1)} Stochastic Env False Stochastic Rew False Sparse Rew False Seed {10 random} ModelType {Tabular, NN} Regress πb False Representation and Function Class For the simpler environments, we use a tabular representation for all the methods. AM amounts to solving for the transition dynamics, rewards, terminal state, etc. through maximum likelihood. FQE, Retrace(λ), Qπ (λ), and TreeBackup are all implemented through dynamics programming with Q tables. MRDR and Q-Reg used the Sherman Morrison [183] method to solve the weighted-least square problem, using a basis which spans a table. In the cases where we needed function approximation, we did not directly fit the dynamics for AM; instead, we fit on the difference in states P (s′ − s|s, a), which is common practice. For the MC environment, we ran experiments with both a linear and NN function class. In both cases, the representation of the state was not changed and remained [position, velocity]. The NN architecture was dense with [16,8,4,2] as the layers. The layers had relu activations (except the last, with a linear activation) and were all initialized with truncated normal centered at 0 with a standard deviation of 0.1. For the pixel-based environments (MC, Enduro), we use a convolutional NN. The architechure is a layer of size 8 with filter (7,7) and stride 3, followed by maxpooling and a layer of size 16 with filter (3,3) and stride 1, followed by max pooling, flattening and a dense layer of size 256. The final layer is a dense layer with the size of the action space, with a linear activation. The layers had elu activations and were all initialized with truncated normal centered at 0 with a standard deviation of 0.1. The layers also have kernel L2 regularizers with weight 1e-6. 136 When using NNs for the IH method, we used the radial-basis function and a shallow dense network for the kernel and density estimate respectively. Datasets Datasets are not included as part of COBS since our benchmark is completely simulation based. To recreate any dataset, select the appropriate choice of environment parameters from the experiments enumerated in Section A.7. Choice of hyperparameters. Many methods require selection of convergence criteria, regularization parameters, batch sizes, and a whole host of other hyperparameters. Often there is a trade-off between computational cost and the accuracy of the method. Hyperparameter search is not feasible in OPE since there is no proper validation (like game score in learning). See Tables A.7, A.8 for a list of hyperparameters that were chosen for the experiments. Figure A.7: Hyperparameters for each model by Environment. Method Parameter Graph TMC MC Pix-MC Enduro Graph-POMDP GW Pix-GW AM Max Traj Len NN Fit Epochs NN Batchsize NN Train size NN Val size NN Early Stop delta T - T - 50 100 32 .8 .2 1e-4 50 100 32 .8 .2 1e-4 - T - T - T 100 25 .8 .2 1e-4 Q-Reg Omega regul. NN Fit Epochs NN Batchsize NN Train size NN Val size NN Early Stop delta 1 - 1 - 80 32 .8 .2 1e-4 80 32 .8 .2 1e-4 80 32 .8 .2 1e-4 1 - 1 - 80 32 .8 .2 1e-4 Convergence ϵ Max Iter NN Batchsize Optimizer Clipnorm 1e-5 - 1e-5 - 1e-4 160 32 1. 1e-4 160 32 1. 1e-4 600 32 1. 1e-5 - 4e-4 50 - 1e-4 80 32 1. Quad. prog. regular. 1e-3 NN Fit Epochs NN Batchsize - 1e-3 - 10001 10001 1024 128 10001 128 1e-3 - 1e-3 1001 128 FQE IH 137 Figure A.8: Hyperparameters for each model by Environment, Cont. Method Parameter Graph TMC MC Pix-MC Enduro Graph-POMDP GW Pix-GW MRDR Omega regul. NN Fit Epochs NN Batchsize NN Train size NN Val size NN Early Stop delta 1 - 1 - 80 1024 .8 .2 1e-4 80 1024 .8 .2 1e-4 80 1024 .8 .2 1e-4 1 - 1 - 80 32 .8 .2 1e-4 R(λ) λ Convergence ϵ Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm .9 1e-3 500 - .9 2e-3 500 - .9 1e-3 80 4 .03 1. - - .9 1e-3 500 - .9 2e-3 50 - .9 1e-3 80 4 .03 1. Qπ (λ) λ Convergence ϵ Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm .9 1e-3 500 - .9 2e-3 500 - .9 1e-3 80 4 .03 1. - - .9 1e-3 500 - .9 2e-3 50 - .9 1e-3 80 4 .03 1. Tree λ Convergence ϵ Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm .9 1e-3 500 - .9 2e-3 500 - .9 1e-3 80 4 .03 1. - - .9 1e-3 500 - .9 2e-3 50 - .9 1e-3 80 4 .03 1. 138 A.8 Additional Supporting Figures Figure A.9: Enduro DM vs IPS. πb is a policy that deviates uniformly from a trained policy 25% of the time, πe is a policy trained with DDQN. IH has relatively low error mainly due to tracking the simple average, since the kernel function did not learn useful density ratio. The computational time required to calculate the multi-step rollouts of AM, Retrace(λ), Qπ (λ), Tree-Backup(λ) exceeded our compute budget and were thus excluded. Figure A.10: MC comparison. N = 256. πb is a uniform random policy, πe is a policy trained with DDQN. 139 Figure A.11: Enduro DM vs HM. πb is a policy that deviates uniformly from a trained policy 25% of the time, πe is a policy trained with DDQN. Figure A.12: Comparison of Direct methods’ performance across horizon and number of trajectories in the Graph environment. Small policy mismatch under a deterministic environment. Figure A.13: (Graph domain) Comparing DMs across horizon length and number of trajectories. Large policy mismatch and a stochastic environment setting. Figure A.14: Comparing DM to DR in a stochastic environment with large policy mismatch. (Graph) 140 Figure A.15: Comparison between FQE, IH, and WIS in a low data regime. For low policy mismatch, IPS is competitive to DM in low data, but as the policy mismatch grows, the top DM outperform. Experiments ran in the Gridworld Environment. Figure A.17: Exact vs Estimated πb . Exact πb = .2−Greedy, πe = .1−Greedy. Min error per class. (Pixel Gridworld, deterministic) Figure A.16: Comparison between IPS methods and IH with dense vs sparse rewards. Per-Decision IPS methods see substantial improvement when the rewards are dense. Experiments ran in the Graph environment with π(a = 0) = .6, πe (a = 0) = .8 Figure A.18: Exact vs Estimated πb . Exact πb =uniform, πe = .1−Greedy. Min error per class. (Pixel Gridworld, deterministic) 141 Figure A.19: Hybrid Method comparison. πb (a = 0) = .2, πe (a = 0) = .8. Min error per class. (Graph-MC) Figure A.20: Hybrid Method comparison. πb (a = 0) = .8, πe (a = 0) = .2. Min error per class. (Graph-MC) Figure A.21: Hybrid Method comparison. πb (a = 0) = .6, πe (a = 0) = .6. Min error per class. (Graph-MC) Figure A.22: Hybrid Method comparison. Exact πb = .2−Greedy, πe = .1−Greedy. Min error per class. (Pixel Gridworld) 142 Figure A.23: Hybrid Method comparison. πb = .8−Greedy(optimal), πe = .1−Greedy(optimal). Min error per class. (Pixel Gridworld) Figure A.24: Class comparison with unknown πb . At first, HM underperform DM because πb is more difficult to calculate leading to imprecise importance sampling estimates. Exact πb = .2−Greedy(optimal), πe = .1−Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) Figure A.25: Class comparison with unknown πb . At first, HM underperform DM because πb is more difficult to calculate leading to imprecise importance sampling estimates. Exact πb = .6−Greedy(optimal), πe = .1−Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) Figure A.26: Class comparison with unknown πb . At first, HM underperform DM because πb is more difficult to calculate leading to imprecise importance sampling estimates. Exact πb =uniform, πe = .1−Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) 143 Figure A.27: AM Direct vs Hybrid comparison for AM. (Gridworld) Figure A.28: FQE Direct vs Hybrid comparison. (Gridworld) Figure A.29: MRDR Direct vs Hybrid comparison. (Gridworld) 144 Figure A.30: Q-Reg Direct vs Hybrid comparison. (Gridworld) Figure A.31: Qπ (λ) Direct vs Hybrid comparison. (Gridworld) Figure A.32: Retrace(λ) Direct vs Hybrid comparison. (Gridworld) 145 Figure A.33: Tree-Backup Direct vs Hybrid comparison. (Gridworld) Figure A.34: DR comparison with πb = .2−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) Figure A.35: WDR comparison with πb = .2−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) Figure A.36: MAGIC comparison with πb = .2−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) Figure A.37: DR comparison with πb = .8−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) 146 Figure A.38: WDR comparison with πb = .8−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) Figure A.39: MAGIC comparison with πb = .8−Greedy(optimal), πe = 1.−Greedy(optimal). (Pixel Gridworld) 147 Appendix B CHAPTER 2 APPENDIX Acronym Term OPE OPO S A P P∗ R X γ π J(π, P ) VπP d0 dP,γ π Off Policy (Policy) Evaluation Off Policy (Policy) Optimization. Also goes by batch off-policy reinforcement learning. State Space Action Space Transition Function True Transition Function Reward Function State-Action Space S × A Discount Factor Policy Performance of π in P Value Function of π with respect to P Initial State Distribution (Discounted) Distribution of State-Action Pairs Induced by Running π in P wπP ν d πb πb Dπb D En [·] E[·] Distribution Shift (wπP (s, a) = Dππ (s,a) ) b Lebesgue measure Behavior state distribution Behavior policy Behavior data (dπb πb ) Dataset containing samples from Dπb P ∗ Empirical approximation using D Exact expectation W V P L Pb ϵH πP∗ RKHS LQR IP MML MLE VAML Distribution Shifts Function Class (e.g. Dππ (s,a) ) Value Function Class (e.g. VπP ∈ V) Model Function Class (e.g. P ∈ P) Model Learning Loss Function Best Model w.r.t L Misspecification Error Optimal Policy in P Reproducing Kernel Hilbert Space Linear Quadratic Regulator Inverted Pendulum Minimax Model Learning (Ours) Maximum Likelihood Estimation Value-Aware Model Learning P,γ d (s,a) dP (s,a) 148 B.1 OPE In this section we explore the OPE results in the order in which they were presented in the chapter. Main Result ∗ Proof for Theorem 4.3.1. Assume (wπP , VπP ) ∈ W × V. Fix some P ∈ P. We use both definitions of J as follows: J(π, P ) − J(π, P ∗ ) = Ed0 [VπP ] − E(s,a)∼dPπ,γ∗ ,r∼R(·|s,a) [r] = E(s,a)∼dPπ,γ∗ [VπP (s) − Er∼R(·|s,a) [r]] + Ed0 [VπP ] − E(s,a)∼dPπ,γ∗ [VπP (s)] = E(s,a)∼dPπ,γ∗ [VπP (s) − Er∼R(·|s,a) [r]] − ∞ X γ t Z dPπ,t (s, a)VπP (s)dν(s, a) Z dPπ,t+1 (s, a)VπP (s)dν(s, a) Z dPπ,t (s̃, ã)P ∗ (s|s̃, ã)π(a|s)VπP (s)dν Z dPπ,t (s, a)P ∗ (s′ |s, a)VπP (s′ )dν ∗ t=1 = E(s,a)∼dPπ,γ∗ [γEs′ ∼P (·|s,a) [VπP (s′ )]] − γ ∞ X γt ∗ t=0 = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )]] − γ ∞ X γ t ∗ t=0 = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )]] − γ ∞ X γt ∗ t=0 = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )]] − γE(s,a)∼dPπ,γ∗ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]] = γE(s,a)∼dPπ,γ∗ [Es′ ∼P (·|s,a) [VπP (s′ )] − Es′ ∼P ∗ (·|s,a) [VπP (s′ )]] ∗  dPπ,γ (s, a)  ∗ ′ = γE(s,a,s )∼Dπb P (·|s,a) [ Ex∼P (·|s,a) [VπP (x)] − VπP (s′ ) ]] Dπb (s, a) ∗   = γE(s,a,s′ )∼Dπb P ∗ (·|s,a) [wπP (s, a) Ex∼P (·|s,a) [VπP (x)] − VπP (s′ ) ]] ∗ = γL(wπP , VπP , P ), where the first equality is definition. The second equality is addition of 0. The third equality is simplification. The fourth equality is change of bounds. The fifth is definition. The sixth is relabeling of the integration variables. The seventh and eighth are simplification. The ninth is importance sampling. The ∗ tenth and last is definition. Since (wπP , VπP ) ∈ W × V then ∗ |J(π, P ) − J(π, P ∗ )| = γ|L(wπP , VπP , P )| ≤γ max w∈W,V ∈V |L(w, V, P )| ≤ γ min max P ∈P w∈W,V ∈V |L(w, V, P )|, where the last inequality holds because P was selected in P arbitrarily. 149 ∗ Now, instead, assume (wπP , VπP ) ∈ W × V. Fix some P ∈ P. Then, similarly, ∗ J(π, P ) − J(π, P ∗ ) = E(s,a)∼dPπ,γ ,r∼R(·|s,a) [r] − Ed0 [VπP ] ∗ ∗ ∗ = E(s,a)∼dPπ,γ [VπP (s)] − Ed0 [VπP ] − E(s,a)∼dPπ,γ [VπP (s) − Er∼R(·|s,a) [r]] = ∞ X γ t=1 ∞ X =γ Z dPπ,t (s, a)VπP (s)dν(s, a) − E(s,a)∼dPπ,γ [VπP (s) − Er∼R(·|s,a) [r]] γt Z dPπ,t+1 (s, a)VπP (s)dν(s, a) − E(s,a)∼dPπ,γ [γEs′ ∼P ∗ (·|s,a) [VπP (s′ )]] Z dPπ,t (s̃, ã)P (s|s̃, ã)π(a|s)VπP (s)dν − γE(s,a)∼dPπ,γ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]] Z dPπ,t (s, a)P (s′ |s, a)VπP (s′ )dν − γE(s,a)∼dPπ,γ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]] t ∗ ∗ ∗ ∗ t=0 =γ ∞ X γ t ∗ ∗ t=0 =γ ∞ X γt ∗ ∗ t=0 ∗ ∗ = γE(s,a)∼dPπ,γ [Es′ ∼P (·|s,a) [VπP (s′ )]] − γE(s,a)∼dPπ,γ [Es′ ∼P ∗ (·|s,a) [VπP (s′ )]] ∗ ∗ = γE(s,a)∼dPπ,γ [Es′ ∼P (·|s,a) [VπP (s′ )] − Es′ ∼P ∗ (·|s,a) [VπP (s′ )]]  dP (s, a)  ∗ ∗ = γE(s,a,s′ )∼Dπb P ∗ (·|s,a) [ π,γ Ex∼P (·|s,a) [VπP (x)] − VπP (s′ ) ]] Dπb (s, a) ∗  ∗  = γE(s,a,s′ )∼Dπb P ∗ (·|s,a) [wπP (s, a) Ex∼P (·|s,a) [VπP (x)] − VπP (s′ ) ]] ∗ = γL(wπP , VπP , P ), ∗ where we follow the same steps as in the previous derivation. Since (wπP , VπP ) ∈ W × V then ∗ |J(π, P ) − J(π, P ∗ )| = γ|L(wπP , VπP , P )| ≤γ max w∈W,V ∈V |L(w, V, P )| ≤ γ min max P ∈P w∈W,V ∈V |L(w, V, P )|, where the last inequality holds because P was selected in P arbitrarily. Sample Complexity for OPE We do not have access to exact expectations, so we must work with Pbn = arg minP maxw,V En [. . .] instead of Pb = arg minP maxw,V E[. . .]. Furthermore, P t J(π, Pb ) requires exact expectation of an infinite sum: Ed0 [ ∞ t=0 γ rt ] where we collect rt by running π in simulation Pb . Instead, we can only estimate P PT t j an empirical average over a finite sum in Pbn : JT,m (π, Pbn ) = m1 m j=1 t=0 γ rt , where each j indexes rollouts starting from s0 ∼ d0 and the simulation is over Pbn . Our OPE estimate is therefore bounded as follows: Theorem B.1.1. [OPE Error] Let the functions in V and W be uniformly bounded by CV and CW respectively. Assume the conditions of Theorem 4.3.1 150 hold and |R| ≤ R̄, γ ∈ [0, 1). Then, with probability 1 − δ, |JT,m (π, Pbn ) − J(π, P ∗ )| ≤ γ min max |L(w, V, P )| P + 4γRn (W, V, P) + w,V 2R̄ T +1 γ 1−γ q 2R̄ q log(2/δ)/(2m) + 4γCW CV log(2/δ)/n, 1−γ where Rn (W, V, P) is the Rademacher complexity of the function class + {(s, a, s′ ) 7→w(s, a)(Ex∼P [V (x)] − V (s′ )) : w ∈ W, V ∈ V, P ∈ P}. Proof for Theorem B.1.1. By definition and triangle inequality, |JT,m (π, Pbn ) − J(π, P ∗ )| = |JT,m (π, Pbn ) − J(π, Pbn ) + J(π, Pbn ) − J(π, P ∗ )| ≤ |JT,m (π, Pbn ) − J(π, Pbn )| + |J(π, Pbn ) − J(π, P ∗ )| . {z | } (a) | {z (B.1) } (b) P Define Vbπ,T (si0 ) ≡ Tt=0 γ t rti for some trajectory indexed by i ∈ N where rti is the reward obtained by running π in P at time t ≤ T starting at si0 . For (a), P |JT,m (π, Pbn ) − J(π, Pbn )| = m m m 1 X 1 X 1 X bn i bn i b b P P (s0 ) − (s0 ) + Vbπ,T Vbπ,∞ Vb Pn (si ) − Ed0 [VπPn ] m i=1 m i=1 m i=1 π,∞ 0 ≤ m m m 1 X 1 X 1 X bn i bn i bn i b P P P (s0 ) − (s0 ) + (s0 ) − Ed0 [VπPn ] Vbπ,T Vbπ,∞ Vbπ,∞ m i=1 m i=1 m i=1 2R̄ q 2R̄ T +1 γ + log(2/δ)/(2m), (B.2) 1−γ 1−γ with probability 1 − δ, where the last inequality is definition of Vbπ,T and Hoeffding’s inequality. ≤ For (b), by Theorem 4.3.1, ∗ |J(π, Pbn ) − J(π, P ∗ )| = γ|L(wπP , V Pn , Pbn )| ≤ γ max |L(w, V, Pbn )| b w,V = γ(max |L(w, V, Pbn )| − max |Ln (w, V, Pbn )| + max |Ln (w, V, Pbn )| w,V w,V w,V − max |L(w, V, Pb )| + max |L(w, V, Pb )|) w,V w,V ≤ γ(2 max ||L(w, V, P )| − |Ln (w, V, P )|| + min max |L(w, V, P )|) w,V,P P w,V q ≤ γ(2R′n (W, V, P) + 2K log(2/δ)/n + min max |L(w, V, P )|) P w,V q ≤ γ(4Rn (W, V, P) + 2K log(2/δ)/n + min max |L(w, V, P )|), P w,V (B.3) 151 where R′n (W, V, P) is the Rademacher complexity of the function class {(s, a, s′ ) 7→ |w(s, a)(Ex∼P [V (x)] − V (s′ ))| : w ∈ W, V ∈ V, P ∈ P}, noting that K = 2Cw CV uniformly bounds |w(s, a)(Ex∼P (·|s,a) [V (x)] − V (s′ ))| (Theorem 8 [19]). Furthermore since absolute value is 1-Lipshitz (by reverse triangle ineq), then R′n < 2Rn (Theorem 12 [19]) where Rn (W, V, P) is the Rademacher complexity of the function class {(s, a, s′ ) 7→ w(s, a)(Ex∼P (·|s,a) [V (x)] − V (s′ ))) : w ∈ W, V ∈ V, P ∈ P}. Altogether, combining (1), (2), (3) we get our result. The first term can be thought of as the estimate under infinite data, the second term as the penalty for using function classes that are too rich, and the remaining terms as the price we pay for finite data/ finite calculations. Misspecification for OPE When the assumptions behind MML do not hold, our method underbounds the true error. The following is the proof for this Proposition. Proof for Prop. 4.3.5. We have shown already that J(π, Pb ) − J(π, P ∗ ) = ∗ γL(wπP , VπP , P ) (= γL((W V )∗ , P )). Therefore, by linearity of L in H, we have |L((W V )∗ , P )| = |L(h, P ) + L((W V )∗ − h, P )| ∀h ∈ H, P ∈ P ≤ |L(h, P )| + |L((W V )∗ − h, P )| ≤ min max |L(h, P )| + |L(h − (W V )∗ , P )| P h ≤ min max |L(h, P )| + max min |L((W V )∗ − h, P )|, P P h h where ϵH = maxP minh |L((W V ) − h, P )|. Therefore |J(π, Pb ) − J(π, P ∗ )| ≤ γ(minP maxh |L(h, P )| + ϵH ), as desired. ∗ Application to the Online Setting and Brief VAML Comparison Algorithm 12 is the prototypical online model-based RL algorithm. In contrast to the batch setting, we allow for online data collection. We require a function called PLANNER, which can take a model Pk and find the optimal solution πk in Pk . 152 Algorithm 12 Online Model-Based RL Require: π0 = πb . PLANNER(·) 1: for k = 0, 1, . . . , K do 2: Collect data Dk by interacting with the true environment using πk . 3: Fit Pk ← arg minP ∈P maxw,V ∈W,V LM M L (w, V, P ) where Dπb = Dk 4: Fit πk ← PLANNER(Pk ) 5: return (PK , πK ) Here we show that MML lower bounds the VAML error in online model-based RL, where VAML is designed. Proposition B.1.2. Let W = {1}. Then min max LM M L (w, V, P )2 ≤ min LV AM L (V, P ), P ∈P w∈W,V ∈V P ∈P for every V, P. Proof. Fix P ∈ P. Then, by definition, LM M L (w, V, P ) = E(s,a,s′ )∼Dπb P ∗ [w(s, a)(Ex∼P (·|s,a) [V (x)] − V (s′ ))] . Since W = {1}, then we can eliminate this dependence and get LM M L (1, V, P ) = E(s,a,s′ )∼Dπb P ∗ [Ex∼P (·|s,a) [V (x)] − V (s′ )]. Explicitly, LM M L (1, V, P )2 Z Z P (x|s, a)V (x)dν(x) − Z Z (P (x|s, a) − P (x|s, a))V (s )dν(x) dν(s, a))2 Z Z (P (x|s, a) − P (x|s, a))V (x)dν(x) =( =( ≤ Z ∗ ′ ′ ∗ ′ V ∈V Z max Z V ∈V ′  2 dν(s, a), Taking the maxV ∈V on both sides and noting maxV any f, V then max LM M L (1, V, P )2 ≤  P (s |s, a)V (s )dν(s ) dν(s, a))2 ∗ R Cauchy Schwarz f (V ) ≤ R maxV f (V ) for 2 (P (x|s, a) − P ∗ (x|s, a))V (x)dν(x) = LV AM L (V, P ). dν(s, a) (B.4) Since we chose P arbitrarily, then Eq B.4 holds for any P ∈ P. In particular, if PbV AM L = arg minP ∈P LV AM L (V, P ) then min max LM M L (1, V, P )2 ≤ max LM M L (1, V, PbV AM L )2 ≤ min LV AM L (V, P ). P ∈P V ∈V V ∈V P ∈P 153 Prop B.1.2 reflects that the MML loss function is a tighter loss in the online model-based RL case than VAML. In a sense, this reflects that MML should be the preferred decision-aware loss function even in online model-based RL. An argument in favor of VAML is that it is more computationally tractable given an assumption that V is the set of linear function approximators. However, if we desire to use more powerful function approximation VAML suffers the same computational issues as MML. In general the pointwise supremum within VAML presents a substantial computational challenge while the uniform supremum from MML is much more mild, can be formulated as a two player game and solved via higher-order gradient descent (see Section B.4). Lastly, VAML defines the pointwise loss with respect to the L2 norm of the difference between P and P ∗ . The choice is justified in that it is computationally friendlier but it is noted that L1 may also be reasonable [59]. We show in the following example that, actually, VAML may not work with a pointwise L1 error. Example B.1.1. Let S = A ∪ B, a disjoint partition of the state space. For simplicity, assume no dependence on actions. Suppose our models P = {Pα }α∈[0,1] take the form Pα (s |s) = ′   α s′ ∈ A  1 − α s′ ∈ B . Suppose also that Pα∗∗ ∈ P for some α∗ ∈ [0, 1]. Let V = {x1s∈A (s) + y1s∈B (s)|x, y < M ∈ R+ } be all bounded piecewise constant value functions with ∥V ∥∞ = M ∈ R+ . Then the empirical VAML loss with L1 pointwise distance does not choose P ∗ when α ̸= 12 and cannot differentiate between P ∗ and any other P ∈ P when α∗ = 12 . MML does not have this issue. Proof. To show this, first fix P ∈ P. Then the empirical VAML loss (in expectation) is given by Es∼P ∗ [max |Ex∼P [V (x)] − V (s)|] V = α max |Ex∼P [V (x)] − V (A)| + (1 − α∗ ) max |Ex∼P [V (x)] − V (B)| ∗ V =α ∗ V max |αx + (1 − α)y − x| + (1 − α ) max |αx + (1 − α)y − y| ∗ x,y∈[0,M ] x,y∈[0,M ] = α∗ max |(α − 1)(x − y)| + (1 − α∗ ) max |α(x − y)| x,y∈[0,M ] = (α∗ |α − 1| + (1 − α∗ )|α|)M. x,y∈[0,M ] 154 If α∗ < .5 then the minimizer of the above quantity is α = 0, if α∗ > .5 then the minimizer is α = 1. Therefore, if α∗ ∈ (0, .5) ∪ (.5, 1) then VAML picks the wrong model α ̸= α∗ . Additionally, in the case that α∗ = .5 then the loss is M2 for every P ∈ P. In this case, VAML with L1 cannot differentiate between any model; all models are perfectly identical. On the other hand, we repeat this process with MML: |Es∼P ∗ [Ex∼P [V (x)] − V (s)]| = |α∗ (Ex∼P [V (x)] − V (A)) + (1 − α∗ )(Ex∼P [V (x)] − V (B))| = |α∗ (αx + (1 − α)y − x) + (1 − α∗ )(αx + (1 − α)y − y)| = |α∗ (α − 1)(x − y) + (1 − α∗ )α(x − y)| = |α − α∗ ||x − y|. Clearly minα∈[0,1] maxx,y∈[0,M ] |α − α∗ ||x − y| = 0 where α = α∗ . We do not have to worry about the choice of norm for MML because we know that the OPE error is precisely LM M L . On the other hand, as shown in the example, this is not the case for VAML. 155 B.2 OPO In this section we explore the OPO results, as presented in the chapter. Main Result Proof for Theorem 4.4.1. Fix some P ∈ P. Through addition of 0, we get J(πP∗ ∗ , P ∗ ) − J(πP∗ , P ∗ ) = J(πP∗ ∗ , P ∗ ) − J(πP∗ ∗ , P ) + J(πP∗ ∗ , P ) − J(πP∗ , P ) + J(πP∗ , P ) − J(πP∗ , P ∗ ). Since πP∗ is optimal in P then J(πP∗ ∗ , P ) − J(πP∗ , P ) ≤ 0 which implies J(πP∗ ∗ , P ∗ ) − J(πP∗ , P ∗ ) ≤ J(πP∗ ∗ , P ∗ ) − J(πP∗ ∗ , P ) + J(πP∗ , P ) − J(πP∗ , P ∗ ) Taking the absolute value of both sides, triangle inequality and invoking Lemma 4.3.1 yields: |J(πP∗ ∗ , P ∗ ) − J(πP∗b, P ∗ )| ≤ 2γ max |L(w, V, Pb )| = 2γ min max |L(w, V, P )| w,V P ∗ ∗ when wπP∗ ∗ , wπPP∗ ∈ W and VπP∗ ∗ , VπPP∗ ∈ V for every P P P ∗ ∗ wπP∗ ∗ , wπPP∗ ∈ W and VπP∗ ∗ , VπPP∗ ∈ V for every P ∈ P. P P w,V ∈ P, or alternatively Sample Complexity for OPO Since we will only have access to the empirical version Pbn rather than Pb , we provide the following bound Theorem B.2.1 (Learning Error). Let the functions in V and W be uniformly bounded by CV and CW respectively. Assume the conditions of Theorem 4.4.1 hold and |R| ≤ R̄, γ ∈ [0, 1). Then, with probability 1 − δ, |J(πP∗b ,P ∗ ) − J(πP∗ ∗ , P ∗ )| ≤ 2γ min max |L(w, V, P )| P n w,V q + 8γRn (W, V, P) + 8γCW CV log(2/δ)/n, where Rn (W, V, P) is the Rademacher complexity of the function class {(s, a, s′ ) 7→ w(s, a)(Ex∼P [V (x)] − V (s′ )) : w ∈ W, P ∈ P, V ∈ V}. Proof for Theorem B.2.1. By Theorem 4.4.1, |J(πP∗b , P ∗ ) − J(πP∗ ∗ , P ∗ )| ≤ 2γ max |L(w, V, Pbn )|. w,V n We have shown in the proof of Theorem 4.3.1 that q max |L(w, V, Pbn )| ≤ min max |L(w, V, P )|+4Rn (W, V, P)+4CW CV log(2/δ)/n. w,V P w,V Combining the two completes the proof. This bound has the same interpretation as in the OPO case; see Section B.1. 156 Misspecification Similarly as in Section B.1, we show the misspecification gap for OPO in the following result. Lemma B.2.2 (OPO Misspecification). Let H ⊂ (S × A × S → R) be ∗ functions on (s, a, s′ ). Denote (W V )∗P ∗ = wπP∗ ∗ (s, a)VπP∗ ∗ (s′ ) and (W V )∗P = P P ∗ wπPP∗ (s, a)VπPP∗ (s′ ). Then: |J(π, Pb ) − J(π, P ∗ )| ≤ 2γ   min max |L(h, P )| + ϵH , P ∈P h∈H (B.5) where ϵH = max(max min |L((W V )∗P ∗ − h, P )|, max min |L((W V )∗P − g, P )|). P ∈P h∈H P ∈P g∈H Proof for Lemma B.2.2. From the proof of Theorem 4.4.1, J(πP∗ ∗ , P ∗ ) − J(πP∗ , P ∗ ) ≤ J(πP∗ ∗ , P ∗ ) − J(πP∗ ∗ , P ) + J(πP∗ , P ) − J(πP∗ , P ∗ ) ∗ ∗ = L(wπP∗ ∗ , VπP∗ ∗ , P ) + L(wπPP∗ , VπPP∗ , P ). P P Using the result from proof of Lemma 4.3.5, ∗ ∗ |L(wπP∗ ∗ , VπP∗ ∗ , P ) + L(wπPP∗ , VπPP∗ , P )| P P ≤ |L(h, P ) + L((W V )∗P ∗ − h, P )| + |L(g, P ) + L((W V )∗P − g, P )| ≤ 2 min max |L(h, P )| + max min |L((W V )∗P ∗ − h, P )| P P h∈H h∈H + max min |L((W V )∗P − g, P )| P g∈H ≤ 2(min max |L(h, P )| + ϵH ), P h∈H where ϵH = max(max min |L((W V )∗P ∗ − h, P )|, max min |L((W V )∗P − g, P )|). g P h Therefore |J(π, Pb ) − J(π, P ∗ )| ≤ 2γ(min P P maxh |L(h, P )| + ϵH ), as desired. 157 B.3 Additional theory In this section, we provide additional results that were not covered in the chapter. Specifically, we show that as we make W, V too rich then the only model with zero loss is P ∗ itself, which may not be in P. Necessary and sufficient conditions for uniqueness of |L(w, V, P )| = 0 When W, V are in L2 then |L| = 0 is uniquely determined: Lemma B.3.1 (Necessary and Sufficient). L(w, V, P ) = 0 for all w ∈ L2 (X , ν) = R R {g : g 2 (x, a)dν(x, a) < ∞}, V ∈ L2 (S, ν) = {f : f 2 (x)dν(x) < ∞} if and only if P = P ∗ wherever Dπb (s, a) ̸= 0. Corollary B.3.2. The same result holds if w · V ∈ L2 (X × S, ν) = {h : R 2 h (x, a, x′ )dν(x, a, x′ ) < ∞}. Proof for Lemma B.3.1 and Corollary B.3.2. We begin with definition 4.5.1 and expand the expectation.   L(w, V, P ) =E(s,a,s′ )∼Dπb (·,·)P ∗ (·|s,a) [w(s, a) Ex∼P (·|s,a) [V (x)] − V (s′ ) ]   = E(s,a)∼Dπb (·,·) [w(s, a) Es′ ∼P (·|s,a) [V (s′ )] − Es′ ∼P (·|s,a) [V (s′ )] ] = Z Dπb (s, a)w(s, a)(V (s′ )(P (s′ |s, a) − P ∗ (s′ |s, a)) dν(s, a, s′ ). (⇒) Clearly if P = P ∗ then L(w, V, P ) = 0. (⇐) For the other direction, suppose L(w, V, P ) = 0. By assumption, w(s, a) can take on any function in L2 (X , ν) and therefore if L(w, V, P ) = 0 then Z V (s′ )(P (s′ |s, a) − P ∗ (s′ |s, a)) dν(s′ ) = 0, (B.6) wherever Dπb (s, a) ̸= 0. Similarly, V (s′ ) can take on any function in L2 (S, ν) and therefore if equation (B.6) holds then P = P ∗ . For the corollary, let (w, V ) ∈ WV take on any function in L2 (X × S, ν). If L(w, V, P ) = 0 then P (s′ |s, a) − P ∗ (s′ |s, a) = 0, as desired. In an RKHS, when the kernel corresponds to an integrally strict positive definite kernel (ISPD), P = P ∗ remains the unique minimizer of the MML Loss: Lemma B.3.3 (Realizability means zero loss even in RKHS). L(w, f, P ) = 0 if and only if P = P ∗ for all (w, V ) ∈ {(w(s, a), V (s′ )) : ⟨wV, wV ⟩Hk ≤ 1, w : X × A → R, V : X → R} in an RKHS with an integrally strict positive definite (ISPD) kernel. 158 Proof for Lemma B.3.3. [208] prove an analogous result and proof here is included for reader convenience. From Mercer’s theorem [145], there exists an 2 orthonormal basis (φj )∞ j=1 of L (X × S, ν) such that RKHS is represented as   WV = w · V = ∞ X j=1 2 bj φj (bj )∞ j=1 ∈ l (N) with ∞ X b2j j=1 µj   < ∞ , where each µj is a positive value since kernel is ISPD. Suppose there exists some P ∈ P such that L(w, V, P ) = 0 for all (w, V ) ∈ WV and P ̸= P ∗ . Then, by taking bj = 1 when (j = j ′ ) and bj = 0 when (j = ̸ j ′ ) for any j ′ ∈ N, we have L(φj , P ) = 0 where we treat w · V as a single input to L. This implies L(w, V, P ) = 0 for all w · V ∈ L2 (X × S, ν) = 0. This contradicts corollary B.3.2, concluding the proof. 159 B.4 Scenarios & Considerations In this section we give proof for the various propositions for the corresponding section in the chapter. Linear Function Classes Proof for Prop. 4.5.1. Given w(s, a)V (s′ ) = ψ(s, a, s′ )T β and P (s′ |s, a) = φ(s, a, s′ )T α then Ln (w, V, P ) = En [Ex∼P [ψ(s, a, x)T β] − ψ(s, a, s′ )T β], = En Z = En [αT  αφ(s, a, x)T ψ(s, a, x)T βdν(x) − ψ(s, a, s′ )T β , Z  φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) β − ψ(s, a, s′ )T β], which is linear in β. L2n (w, V, P ) = 0 is achieved through En [αT Z  φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) − ψ(s, a, s′ )T ] = 0. Thus, b T = En [ψ(s, a, s′ )T ]En α hR Z −1 φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) , i assuming En φ(s, a, s′ )ψ(s, a, s′ )T dν(s′ ) is full rank. Taking the transpose completes the proof. Proof for Prop. 4.5.2. We begin with φ(s, a, s′ ) = e(s,a,s′ ) , the (s,a,s’)-th standard basis vector and ψ = φ. Then X(s, a) = ( X x∈S φ(s, a, x)φ(s, a, x)T )i,j =   1 i = s|A||S| + a|S|, i = j  0 otherwise . Notice that X(s, a) is a diagonal matrix and is the discrete counter-part to R P φ(s, a, s′ )ψ(s, a, s′ )T dν(x). Therefore, En [X(s, a)] = N1 (s,a,s′ )∈D X(s, a), which is a diagonal matrix of the average number of times (s, a) appears in the dataset D. Similarly, En [φ(s, a, s′ )] is the average number of times that (s, a, s′ ) appears in the dataset D. Hence, by Prop 4.5.1, #{(s, a, s′ ) ∈ D} b s,a,s′ = α . #{(s, a, x) ∈ D : ∀x ∈ S} b =α b s,a,s′ , as desired. Therefore P (s′ |s, a) = φ(s, a, s′ )T α LQR In order to provide proof that MML gives the LQR-optimal solution, we begin with a few Lemmas. First, we show that the value function is quadratic. 160 Lemma B.4.1 (Value Function is Quadratic). Let st+1 = Ast + Bat + w with 2 w ∼ N (0, σ ∗2 I) be the dynamics, πK (a|s) = −Ks + wK where wK ∼ N (0, σK I) T be the policy. Let γ ∈ (0, 1] be the discount factor. Then V (s) = s U s + q where U = Q + K T RK + γ(A − BK)T U (A − BK) 1 2 (σ 2 tr(R) + γσK tr(B T U B) + γσ ∗2 tr(U )). q= 1−γ K Proof for Lemma B.4.1. The value function is given by: xT U x + q = xT Qx + EN (−Kx,σK2 I) [uT Ru + γEN (Ax+Bu,σ∗2 I) [V (s′ )]] = xT Qx + EN (−Kx,σK2 I) [uT Ru + γ(Ax + Bu)T U (Ax + Bu) + γq + γσ ∗2 tr(U )] 2 = xT Qx + xT K T RKx + σK tr(R) + γxT (A − BK)T J(A − BK)x 2 + γσK tr(B T U B) + γq + γσ ∗2 tr(U ). Thus, the quadratic terms satisfies: U = Q + K T RK + γ(A − BK)T U (A − BK), and the linear term satisfies: 1 2 (σ 2 tr(R) + γσK tr(B T U B) + γσ ∗2 tr(U )). q= 1−γ K The final value is given by: J(π, P ∗ ) = EN (s0 ,σ02 I) [U ] = sT0 U s0 + q + σ02 tr(U ). Existence and uniqueness of U, q is heavily studied [22]. Under the same assumptions as Lemma B.4.1, we simplify L to: Lemma B.4.2 (LQR Loss Simplified). In addition to the assumptions of Lemma B.4.1, let d0 = s0 + wd0 where wd0 ∼ N (0, σd20 I) be the initial state distribution. Let P = As + Ba ∈ P where A ∈ Rn×n , B ∈ Rn×k and (A, B) is controllable. Let K ∈ Rk×n represent all linear policies and U ∈ Sn+ be all symmetric positive semi-definite matrices. min max |L(w, V, P )| P w,V = min max A,B K,U X γ i [sT0 (A∗ − B ∗ K)iT ∆(A∗ − B ∗ K)i s0 i 2 + tr(∆Σi )] + σK tr(B T U B − B ∗T U B ∗ ) − σ ∗2 tr(U ), 161 where ∆ = (A − BK)T U (A − BK) − (A∗ − B ∗ K)T U (A∗ − B ∗ K) and Σi = σ ∗ (I + . . . + F i−1 F (i−1)T ) + σK (B ∗ B ∗T + . . . + F i−1 B ∗ B ∗T F (i−1)T ) + σ0 F i F iT for i > 0 and Σ0 = σ0 I, F = A∗ − B ∗ K. Proof for Lemma B.4.2. We first show that the evolution of dynamics P ∗ under gaussian noise, with a linear gaussian controller is a gaussian mixture P ∗ ∗ i ∗ i−1 (i−1)T F ) + σK (B ∗ B ∗T + i N ((A − B K) s0 , Σi ), where Σi = σ (I + . . . + F . . . F i−1 B ∗ B ∗T F (i−1)T ) + σ0 F i F iT for i > 0 and Σ0 = σ0 I, F = A∗ − B ∗ K. It’s clear s0 ∼ N (s0 , σ02 I), the base case. Suppose for induction sn ∼ N ((A∗ − B ∗ K)n s0 , Σn ) holds for some n ≥ 0. Then sn+1 = A∗ sn + B ∗ (−Ksn + wK ) + w∗ = (A∗ − B ∗ K)sn + B ∗ wk + w∗ ∼ N ((A∗ − B ∗ K)n+1 s0 , (A∗ − B ∗ K)Σn (A∗ − B ∗ K)T + B ∗ B ∗T + σ ∗ I) = N ((A∗ − B ∗ K)n+1 s0 , Σn+1 ), completing the inductive step. Every step st is gaussian, therefore ∗ dPπ,γ (s, a) = ∞ X 2 γ i N (s; F i s0 , Σi )N (a; −Ks, σK I) (B.7) i=0 dP ∗ is a gaussian mixture. Let w = π,γ . We know V is quadratic, given by U ∈ S+n . D For notational convenience, for matrix Z and vectors x,y define ⟨x, y⟩Z ≡ xT Zy and similarly ∥x∥Z ≡ ⟨x, x⟩Z = xT Zx. Therefore, min max L(w, V, P ) = min max E(s,a)∼D [w[EP [V ] − EP ∗ [V ]]] P w,V A,B w,V = min max E(s,u)∼D [w[∥As + Bu∥U − ∥A∗ s + B ∗ u∥U − σ ∗2 tr(U )]] A,B w,U = min max EP γ i N ((A∗ −B ∗ K)i s0 ,Σi ) [Eu∼N (−Ks,σK2 I) [∥As + Bu∥U A,B K,U i − ∥A∗ s + B ∗ u∥U − σ ∗2 tr(U )]] = min max EP γ i N ((A∗ −B ∗ K)i s0 ,Σi ) [∥(A − BK)s∥U − ∥(A∗ − B ∗ K)s∥U A,B K,U i 2 2 + σK tr(B T U B) − σK tr(B ∗T U B ∗ ) − σ ∗2 tr(U )] = min max EP γ i N ((A∗ −B ∗ K)i s0 ,Σi ) [sT [∆(A, B, A∗ , B ∗ , U, K)]s A,B K,U i 2 tr(B T U B − B ∗T U B ∗ ) − σ ∗2 tr(U )] + σK = min max A,B K,U X γ i [∥(A∗ − B ∗ K)i s0 ∥∆ + tr(∆Σi )] i 2 + σK tr(B T U B − B ∗T U B ∗ ) − σ ∗2 tr(U ), where ∆ = (A − BK)T U (A − BK) − (A∗ − B ∗ K)T U (A∗ − B ∗ K). 162 First, Lemma B.4.2 supposes that there is model mismatch P ∗ ̸∈ P since P are deterministic simulators and P ∗ is stochastic. Second, we notice that K takes the position of w, which is to say that the policy K directly specifies w, as expected. We will need the previous two results in the experiments. We may now prove Prop 4.5.3 that says MML yields the true parameters of LQR in expectation: Proof for Prop 4.5.3. Consider two linear, controllable systems with parameters P1 = (A1 , B1 ) and P2 = (A2 , B2 ). Then there exists a controller K that stabilizes P1 (i.e, J(P1 , K) < ∞) but destabilizes P2 (i.e, J(P2 , K) = ∞). We show this by analyzing the characteristic polynomial of both A1 − B1 K and A2 − B2 K. There exists an invertible matrix T1 , T2 that put (A1 , B1 ), (A2 , B2 ) into controllable canonical forms (CCF), respectively [22]. Thus, we will assume, wlog, that (Ã1 , B̃1 ), (Ã2 , B̃2 ) are already in CCF. Hence, 0    0   .. Ã1 =   .  1 0 .. . 0 1 .. . ... ... 0 0 .. . 0 0 0 1 −a0 −a1 −a2 . . . −an−1    0         ,         0    ..  B̃1 =  , .   0   1 and 0    0  . Ã2 =   ..     0  1 0 .. . 0 1 .. . ... ... 0 0 .. . 0         ,         0   . B̃2 =   ..  .     0   0 0 1 −b0 −b1 −b2 . . . −bn−1 1 We will find a controller in the form K = K1 T1 = K2 T2 for some K1 , K2 for T1 , T2 that put the systems into CCF. Consider a desired characteristic polynomial of f (s) = (s + ϵ)n−1 (s + λ) for ϵ, λ ∈ R+ (> 0). This polynomial has eigenvalues equal to −ϵ, −λ and therefore a system with this polynomial is asymptotically stable (converges to 0 exponentially fast). Take K1 = [k1,0 , k1,1 , . . . , k1,n−1 ]. Then det(sI − (Ã1 − B̃1 K1 )) = sn + (an−1 + k1,n−1 )sn−1 + · · · + (a0 + k1,0 ). By se     n−1 λ + ϵ ϵn−1−i − ai then det(sI − (Ã1 − B̃1 K1 )) = f (s). lecting k1,i = n−1 i i−1 Hence, (Ã1 , B̃1 ) is asymototically stable with eigenvalues −λ, −ϵ for any λ, ϵ strictly positive. Therefore K = K1 T1 makes the system (A1 , B1 ) asymptotically stable. Now we consider K2 = K1 T1 T2−1 . Let us denote T1 T2−1 = T which is also 163 invertible since T1 , T2 are invertible. Then by taking the last term of det(sI − Q (Ã2 − B̃2 K2 )), we can examine the product n−1 i=0 λi of the eigenvalues of the P closed loop system Ã2 − B̃2 K2 . Namely, b0 + n−1 i=0 k1,i Ti,n is the product of eigenvalues. We may simplify this via some algebra as follows: n−1 Y λi = b 0 + i=0 = b0 + = b0 − n−1 X i=0 n−1 X i=0 n−1 X i=0 k1,i Ti,n n−1 n−1 λ+ ϵ ϵn−1−i − ai i i−1 ! Ti,n ai + n−1 X Ti,n ! n−1 X n − 1 n−i n − 1 n−1−i ϵ +λ Ti,n ϵ i−1 i i=0 ! i=0 | ! ! {z ! } b̄ | {z c } = b̄ + λc. We may select ϵ > 0 so that c ̸= 0 otherwise Ti,n = 0 for all i which would Q contradict invertibility of T . Therefore n−1 i=0 λi is linear in λ. By driving Qn−1 λ → ∞, then | i=0 λi | → ∞ is unbounded. Select λ so that |b̄ + λc| > 1. By the pigeonhole principle, at least one of the eigenvalues of Ã2 − B̃2 K2 must have a magnitude greater than 1 and therefore the system is unstable. Therefore the controller K2 T2 = K1 T1 T2−1 T2 = K1 T1 = K makes the system (A2 , B2 ) unstable. Hence, K simultaneously stabilizes (A1 , B1 ) but destabilizes (A2 , B2 ). According to Lemma B.4.2, when (A, B) = (A∗ , B ∗ ) then for any K: max L((A, B), K, U ) = max |σ ∗2 tr(U )| < ∞, U U since U are bounded by assumption. Furthermore, we have just shown that there always exists a K that destabilizes any controller (A, B) ̸= (A∗ , B ∗ ) while stabilizing (A∗ , B ∗ ). Therefore: max L((A, B), K, U ) = ∞, K,U for any system (A, B) ̸= (A∗ , B ∗ ). Therefore: min max L((A, B), K, U ) = (A∗ , B ∗ ). (A,B) K,U It is well known that ordinary least squares is a consistent estimator when the noise is exogenous, as it is here. Therefore the maximum likelihood solution also yields (A∗ , B ∗ ) in expectation. RKHS & Practical Implementation Since P ∈ P is a stochastic model in general, then the inner expectation of the loss in def (4.5.1) over P involves sampling x from P (·|s, a) and computing the 164 empirical average of V (x). In general this can be computationally demanding if S is high dimensional and P does not have a closed form, requiring MCMC estimates or variational estimates [139, 73]. However, in practice, most parametrizations of models use nice distributions, such as gaussians, from which sampling is efficient. This issue is similarly present in other decision-aware literature [e.g., 59]. The estimator based on Eq (4.8) requires solving a minimax problem which is often computationally challenging. One approach might be to set-up GAN-style neural networks and use higher order gradient descent [74, 178]. If we have access to a kernel, say radial basis function (RBF), then the inner maximization over w, V has a closed form when W × V correspond to a reproducing kernel Hilbert space (RKHS), HK with kernel K. In particular, in similar spirit to [134, 62, 208] we have Proposition B.4.3 (Closed form exists in RKHS). Assume WV = {(w(s, a), V (s′ )) : ⟨wV, wV ⟩HK ≤ 1, w : X → R, V : S → R}. Let ⟨·, ·⟩HK be an inner product on HK satisfying the reproducing kernel property w(s, a)V (s′ ) = ⟨wV, K((s, a, s′ ), ·)⟩HK . The term max(w,V )∈WV L(w, V, P )2 has a closed form: " max L(w, V, P ) = E(s,a,s′ )∼Dπb P ∗ ,(s̃,ã,s̃′ )∼Dπb P ∗ 2 (w,V )∈WV Ex∼P,x̃∼P [K((s, a, x), (s̃, ã, x̃))] − 2Ex∼P [K((s, a, x), (s̃, ã, s̃′ ))] # + K((s, a, s′ ), (s̃, ã, s̃′ )) . Proof for Prop B.4.3. Recall that by the reproducing property of kernel K in the RKHS space HK then ⟨f, K⟩HK for any f ∈ HK . Starting from Def 4.5.1,   L(w, V, P )2 = E(s,a,s′ )∼Dπb (·,·)P ∗ (·|s,a) [w(s, a) Ex∼P (·|s,a) [V (x)] − V (s′ ) ]2 = E(s,a,s′ ,x)∼Dπb (·,·)P ∗ (·|s,a)P (·|s,a) [w(s, a)V (x) − w(s, a)V (s′ )]2 = E(s,a,s′ ,x)∼Dπb (·,·)P ∗ (·|s,a)P (·|s,a) [⟨wV, K((s, a, x), ·)⟩Hk − ⟨wV, K((s, a, s′ ), ·)⟩Hk ]2 = ⟨wV, (wV )∗ ⟩2Hk where (wV )∗ (·) = E(s,a,s′ ,x)∼Dπb (·,·)P ∗ (·|s,a)P (·|s,a) [K((s, a, x), ·) − K((s, a, s′ , ·)]. By Cauchy-Schwarz and the fact that wV is within a unit ball, then max L(w, f, V )2 = max ⟨wV, (wV )∗ ⟩2Hk = ∥(wV )∗ ∥2 = ⟨(wV )∗ , (wV )∗ ⟩Hk . w,V ∈WV w,V ∈WV 165 Expanding, max L(w, f, V )2 w,V ∈WV = ⟨(wV )∗ , (wV )∗ ⟩Hk = ⟨E(s,a,s′ ,x)∼Dπb (·,·)P ∗ (·|s,a)P (·|s,a) [K((s, a, x), ·) − K((s, a, s′ , ·)], E(s̃,ã,s̃′ ,x̃)∼Dπb (·,·)P ∗ (·|s̃,ã)P (·|s̃,ã) [K((s̃, ã, x̃), ·) − K((s̃, ã, s̃′ , ·)]⟩Hk = *Z Z Dπb (s, a)P ∗ (s′ |s, a)P (x|s, a)(K((s, a, x), ·) − K((s, a, s′ ), ·)), + Dπb (s̃, ã)P (s̃ |s̃, ã)P (x̃|s̃, ã)(K((s̃, ã, x̃), ·) − K((s̃, ã, s̃ ), ·)), ∗ ′ ′ Hk = Z Dπb (s, a)P ∗ (s′ |s, a)P (x|s, a)Dπb (s̃, ã)P ∗ (s̃′ |s̃, ã)P (x̃|s̃, ã) × ⟨K((s, a, x), ·) − K((s, a, s′ ), ·), K((s̃, ã, x̃), ·) − K((s̃, ã, s̃′ , ·)⟩Hk . By linearity of the inner product, the reproducing kernel property we get: max (w,V )∈WV L(w, f, V )2 = E(s,a,s′ ,x)∼Dπb P ∗ P,(s̃,ã,s̃′ ,x̃)∼Dπb P ∗ P [K((s, a, x), (s̃, ã, x̃)) − K((s, a, x), (s̃, ã, s̃′ )) − K((s, a, s′ ), (s̃, ã, x̃)) + K((s, a, s′ ), (s̃, ã, s̃′ ))] = E(s,a,s′ ,x)∼Dπb P ∗ P,(s̃,ã,s̃′ ,x̃)∼Dπb P ∗ P [K((s, a, x), (s̃, ã, x̃)) − 2K((s, a, x), (s̃, ã, s̃′ )) + K((s, a, s′ ), (s̃, ã, s̃′ ))], where for the last equality we used the fact that K is symmetric. 166 B.5 Experiments Environment Descriptions LQR The LQR domain is a 1D stochastic environment with true dynamics: P ∗ (s′ |s, a) = s−.5a+w∗ where w∗ ∼ N (0, .012 ). We let x0 ∼ N (1, .12 ). The reward function is R(s, a) = −(s + a) and γ = .9. We use a finite class P consisting of all deterministic models P = {Px (s′ |s, a) = (1 + x/10)s − (.5 + x/10)a|x ∈ [0, M ]} where we vary M ∈ {2, 3, . . . , 19}. We write (A∗ , B ∗ ) = P0 (s′ |s, a) = A∗ s + B ∗ a, the deterministic version of P ∗ . Cartpole We use the standard Cartpole benchmark (OpenAI, [30]). The state space is a tuple (x, ẋ, θ, θ̇) representing the position of the cart, velocity of the cart, angle of the pole and angular velocity of the pole, respectively. The action space is discrete given by pushing the car to the left or pushing the car to the right. We add N (0, .0012 ) Gaussian noise to each component of the state to make the dynamics stochastic. We consider the infinite horizon setting with γ = .98. The reward function is modified to be a function of angle and location R(s, θ) = (2 − θ/θmax ) ∗ (2 − s/smax ) − 1) rather than 0/1 to make the OPE problem more challenging. Inverted Pendulum (IP) We consider the infinite horizon setting with γ = .98. The state space is a tuple (θ, θ̇) representing the angle of the pole and angular velocity of the pole, respectively. The action space A = R is continuous representing a clockwise or counterclockwise force. The reward function is a clipped quadratic function R([θ, θ̇], a) = min(((θ + π) mod 2π − π)2 + .1θ̇2 + .001u2 , 100). This IP environment has a Runge-Kutta(4) integrator [51] rather than Forwrd Euler and, thus, produces more realistic data. The mass of the rod is .25 and the length .5. Experiment Descriptions LQR OPE/OPO OPE. We aim to evaluate π(a|s) = N (1.3s, .12 ). We ensure VπP ∈ V for all ∗ P ∈ P by solving the equations in Lemma B.4.1. We ensure WπP ∈ W using 167 Equation (B.7). We derive 1-d equations for VAML analogous to Lemma B.4.2). Finally, we know MLE gives (A∗ , B ∗ ) in expectation (see Prop 4.5.3). Metric: We compute |(J(π, Pb ) − J(π, P ∗ ))|, the OPE error. OPO. Similarly as in OPE, we ensure that all MML realizability assumptions hold. This means as we increase P then we have to increase the sizes of both W and V now instead of just V as in OPE. Once again MLE gives (A∗ , B ∗ ) in expectation (see Prop 4.5.3) and we evaluate VAML using equations analogous to those in Lemma B.4.2). With this, we produce Figure B.1 (right). By increasing P, we also have more policies {πP∗ }P ∈P we may consider. Instead of selecting one for OPE, for each π ∈ {πP∗ }P ∈P we calculate the OPE error. We aggregate across all {πP∗ }P ∈P by taking the average of the OPE errors and the worst-case, which can be seen in Figure B.1 (left). Metric: We compute |(J(πP∗b, Pb ) − J(πP∗ ∗ , P ∗ ))|, the OPO error. Note: All calculations in LQR OPE/OPO are in expectation so no error bars need be included. Verifiability. With the same setup as in OPE, now randomly sample 100k points in the interval [−3, 3] × [−3, 3], which is the support of the LQR system. We rerun the same experiment as in OPE except now we add w ∼ N (0, ϵ) noise to V ∈ V where ϵ ∈ {0, .2, . . . , .8, 1}. We evaluate the error |(J(π, Pb ) − J(π, P ∗ ))| over the 100k samples rather than in expectation as before. We run 5 seeds and present the mean over the seeds with standard error. We smooth the resulting mean with a moving average filter of size 3. The result can be seen in Figure 4.2 (right). Cartpole OPE Each P ∈ P takes the form s′ ∼ N (µ(s, a), σ(s, a)), where a NN outputs a mean, and logvariance representing a normal distribution around the next state. Each model has a two hidden layers and with 64 units each and ReLU activation with final linear layer. We generate the behavior and target policy using a near-perfect DDQN-based policy Q with a final softmax layer and adjustable parameter τ : π(a|s) ∝ exp(Q(s, a)/τ ). The behavior policy has τ = 1, while the target policy has τ = 1.5. We truncate all rollouts at 1000 time steps and we calculate the true expected value using the Monte Carlo average of 10000 rollouts. 168 We model the class WV as a RKHS as in Lemma B.4.3 with an RBF kernel. We do the same for VAML. The RKHS kernel we use for MML and VAML is given by K(s, a, s′ ) = K1 (s)K2 (a)K3 (s′ ) and K3 (s′ ) respectively where Ki are Gaussian Radial Basis Function (RBF) kernels with a bandwidth equal to the median of the pair-wise distances for each coordinate (s, a, s′ independently) over the batch. For MML, we sample from P a total of 5 times and take the empirical mean to calculate the expectation over P for the RKHS formula given in B.4.3. We run 20000 batches of size 128 and normalize the data over the batch. Our learning rate is 10−3 and we use Adam [107] optimizer. The estimate we use is the mean over the last 10 batches. We run 5 random seeds per dataset size, and plot the log-relative MSE with standard error in Figure 4.3. Note: These hyperparameters remain the same across the different loss functions. Metric: We compare the methods using the log-relative MSE metric: (J(π, Pb ) − J(π, P ∗ ))2 log( ), (J(πb , P ∗ ) − J(π, P ∗ ))2 which is negative when the OPE estimate J(π, Pb ) is superior to the on-policy estimate J(πb , Pb ). The more negative, the better the estimate. To calculate J(π, Pb ) we run 100 trajectories in Pb and take the mean. Inverted Pendulum OPO We generate the behavior data using a noisy feedback-linearized controller: πb (a|s) is uniformly random with probability .3 and is a feedback-linearized LQR controller (FLC) with probability .7 where we use the FLC corresponding to LQR matrices Q = 2I2×2 , R = I2×2 . We truncate all rollouts at 200 time steps. We fit 4 feed-forward neural networks representing P1 , . . . , P4 where each is a deterministic model with two layers of 16 weights and a Tanh activation followed with Linear. We use Adam [107] optimizer with 10−3 as the learning rate. Using different batches of size 64 on each Pi and perform 5000 iterations for each model. The RKHS kernel we use for MML and VAML is given by K(s, a, s′ ) = K1 (s)K2 (a)K3 (s′ ) and K3 (s′ ) respectively where Ki are Gaussian Radial Basis Function (RBF) kernels with a bandwidth equal to 1. 169 For MML, we only sample from P once to calculate the expectation over P for the RKHS formula given in B.4.3, since P is deterministic. Now we have P (s′ |s, a) = 14 4i=1 Pi (s′ |s, a). We calculate α = M edian({∥Pj (s, a)− s′ ∥2 : j ∈ [1, . . . , 4], (s, a, s′ ) ∈ X ⊂ D}) where X is 10000 random samples from the dataset. We form an α-USAD (see MOREL Section B.5) and construct a pessimistic MDP (P̃ , R̃) (see Section B.5). We use PPO as our policy optimizer with the default settings from [169]. We run PPO three times in the pessimistic MDP and take the policy that performs the best and report its performance. We keep track of the running maximum as we increase the dataset size. We plot the mean of the running maximums over the five seeds including standard error bars in Figure 4.4. P Note: These hyperparameters remain the same across the different loss functions. Metric: We look at the performance J(πP∗b, P ∗ ) of a policy and compare it to π ∗ , learned from PPO. To calculate J(πP∗b, P ∗ ) we run 100 trajectories in P ∗ and take the mean. MOREL We give a brief explanation of MOREL [106] and its construction. The objective of MOREL is to make sure that the policy we learn does not take advantage of the errors in the simulator P . If there are errors in P then a policy may think the agent can perform a particular state transition (s, a, s′ ) and R(s′ , a′ ) has high reward for some action a′ . However, it’s possible that such a transition (s, a, s′ ) may not occur in the true environment. Therefore, we modify our model P (s′ |s, a) in the following way:   Terminate episode P̃ (s |s, a) =  P (s′ |s, a) ′ U α (s, a) = 1 otherwise , where U α (s, a) = 1 if maxi∈{1,2,3,4} ∥Pi (s′ |s, a) − P (s′ |s, a)∥ ≥ α, otherwise 0. In other words, we’ve modified the transition dynamics so that we do not trust our model P unless all the Pi are in agreement. We also modify our reward to be  R̃(s, a) =  -100 U α (s, a) = 1  R(s, a) otherwise , 170 where −100 is chosen this value is well below any reward that the Inverted Pendulum environment generates. Similarly, we penalize our policy for entering a state where we are uncertain. Together, this creates a pessimistic MDP. Additional Experiments 0.10 0.24 0.20 0.18 0.16 0.14 0.12 MML means MML maxs MLE, VAML, (A , B ) means MLE, VAML, (A , B ) maxs 0.10 0.08 2 4 6 8 10 12 Number of Models, | | ( = | | = | |) 14 OPO Error, |J( P* , P) J( P** , P * )| OPE Error, |J( , P) J( , P * )| 0.22 MML MLE, VAML, (A , B ) 0.08 0.06 0.04 0.02 0.00 2 4 6 8 10 12 Number of Models, | | ( = | | = | |) 14 Figure B.1: (LQR) As we increase |W |, |V | then MML is forced to be robust to too many OPE problems and settles for the system (A∗ , B ∗ ) since this is the only system robust to the most OPE problems. In the experiments for Figure B.1, we consider what happens when we satisfy the realizability conditions for OPO. As we increase |P|, we must also increase |W|, |V| because each P ∈ P induces an optimal policy πP∗ to which we have ∗ to make sure wπPP∗ ∈ W and VπPP∗i ∈ V for ∀Pi ∈ P. In a sense, we are adding more OPE problems for MML to be robust to. In particular, we now have more policies {πP∗ }P ∈P to consider. As described earlier, for each π ∈ {πP∗ }P ∈P we calculate the OPE error. We aggregate across all {πP∗ }P ∈P by taking the average of the OPE errors and the worst-case, which can be seen in Figure B.1 (left). We plot the OPO error in Figure B.1 (right). What we see is that while |P| is small, MML is able to be robust to a certain number of OPE problems. But as we increase the number of OPE problems the average and max error increases until all methods select the same model, which is the OPO-optimal model, (A∗ , B ∗ ). 171 Appendix C CHAPTER 3 APPENDIX C.1 Preliminaries to Analysis of Fitted Q Evaluation (FQE) In this section, we set-up necessary notations and definitions for the theoretical analysis of FQE. To simplify the presentation, we will focus exclusively on weighted ℓ2 norm for error analysis. With the definitions and assumptions presented in this section, we will present the sample complexity guarantee of Fitted-Q-Evaluation (FQE) in appendix C.2. While it is possible to adapt proofs from related algorithms [150, 12] to analyze FQE, in the next two sections we show improved convergence rate from O(n−4 ) to O(n−2 ), where n is the number of samples in data set D. Our notation for Q function is similar to the RL literature — the only difference is that the optimal policy minimizes Q(s, a) instead of maximizing. We denote the bound on the value function as C (alternatively if the single timestep cost c is bounded by c, then C = 1−γ ). Bellman operators The Bellman optimality operator T : B(X ; C) 7→ B(X ; C) as: (TQ)(s, a) = c(s, a) + γ Z min Q(s′ , a′ )P (ds′ |s, a). (C.1) S a′ ∈A The optimal value functions are defined as usual by C ∗ (s) = sup C π (s) and π Q∗ (s, a) = sup Qπ (s, a) ∀(s, a) ∈ X . π For a given policy π, the Bellman evaluation operator Tπ : B(X ; C) 7→ B(X ; C) as: (T Q)(s, a) = c(s, a) + γ π Z S Q(s′ , π(s′ ))P (ds′ |s, a). (C.2) It is well known that Tπ Qπ = Qπ , a fixed point of the Tπ operator. Data distribution and weighted ℓ2 norm Denote the state-action data generating distribution as µ, induced by some data-generating (behavior) policy πD , that is, (si , ai ) ∼ µ for (si , ai , s′i , ci ) ∈ D. Note that data set D is formed by multiple trajectories generated by πD . For 172 each (si , ai ), we have s′i ∼ P (·|si , ai ) and ci = c(si , ai ). For any (measurable) function f : X 7→ R, define the µ-weighted ℓ2 norm of f as ∥f ∥2µ = R R 2 2 X f (s, a) µ(ds, da) = X f (s, a) µx (ds)πD (a|ds). Similarly for any other stateR action distribution ρ, ∥f ∥2ρ = X f (s, a)2 ρ(ds, da). Inherent Bellman error FQE depends on a chosen function class F to approximate Q(s, a). To express how well the Bellman operator Tg can be approximated by a function in the policy class F, when Tg ∈ / F, a notion of distance, known as inherent Bellman error was first proposed by [147] and used in the analysis of related ADP algorithms [150, 148, 11, 12, 119, 118, 120, 140]. Definition C.1.1 (Inherent Bellman Error). Given a function class F and a chosen distribution ρ, the inherent Bellman error of F is defined as: dF = d(F, TF) = sup inf ∥f − Th∥ρ , h∈F f ∈F where ∥·∥ρ is the ρ−weighted ℓ2 norm and T is the Bellman optimality operator defined in (C.1). To analyze FQE, we will form a similar definition for the Bellman evaluation operator Definition C.1.2 (Inherent Bellman Evaluation Error). Given a function class F and a policy π, the inherent Bellman evaluation error of F is defined as: dπF = d(F, Tπ F) = sup inf ∥f − Tπ h∥ρπ , h∈F f ∈F where ∥·∥ρπ is the ℓ2 norm weighted by ρπ . ρπ is defined as the state-action distribution induced by policy π, and Tπ is the Bellman operator defined in (C.2) Concentration coefficients Let P π denote the operator acting on f : X 7→ R such that (P π f )(s, a) = R ′ ′ ′ ′ π S f (s , π(s ))P (s |s, a)ds . Acting on f (e.g., approximates Q), P captures the transition dynamics of taking action a and following π thereafters. The following definition and assumption are standard in the analysis of related approximate dynamic programming algorithms [118, 150, 11]. As approximate value iteration and policy iteration algorithms perform policy update, the new policy at each round will induce a different stationary state-action distribution. 173 One way to quantify the distribution shift is the notion of concentrability coefficient of future state-action distribution, a variant of the notion introduced by [147]. Definition C.1.3 (Concentration coefficient of state-action distribution). Given data generating distribution µ ∼ πD , initial state distribution d0 . For m ≥ 0, and an arbitrary sequence of stationary policies {πm }m≥1 let d(d0 P π1 P π2 . . . P πm ) βµ (m) = sup . dµ π1 ,...,πm ∞ (βµ (m) = ∞ if the future state distribution d0 P π1 P π2 . . . P πm is not absolutely continuous w.r.t. µ, i.e, d0 P π1 P π2 . . . P πm (s, a) > 0 for some µ(s, a) = 0) Assumption C.1. βµ = (1 − γ)2 P m≥1 mγ m−1 βµ (m) < ∞. Combination Lock Example. An example of an MDP that violates Assumption C.1 is the “combination lock” example proposed by [109]. In this finite MDP, we have N states S = {1, 2, . . . , N }, and 2 actions: going L or R. The initial state is s0 = 1. In any state x, action L takes agent back to initial state s0 , and action R advances the agent to the next state x + 1 in a chain fashion. Suppose that the reward is 0 everywhere except for the very last state N . One can see that for an MDP such that any behavior policy πD that has a bounded from below probability of taking action L from any state x, i.e., πD (L|s) ≥ ν > 0, then it takes an exponential number of trajectories to learn or evaluate a policy that always takes action R. In this setting, we can see that the concentration coefficient βµ can be designed to be arbitrarily large. Complexity measure of function class F Definition C.1.4 (Random L1 Norm Covers). Let ϵ > 0, let F be a set of functions S 7→ R, let sn1 = (s1 , . . . , sn ) be n fixed points in S. Then a collection of functions Fϵ = {f1 , . . . , fN } is an ϵ-cover of F on sn1 if n n 1X 1X ∀f ∈ F, ∃f ′ ∈ Fϵ : | f (si ) − f ′ (si )| ≤ ϵ. n i=1 n i=1 The empirical covering number, denote by N1 (ϵ, F, sn1 ), is the size of the smallest ϵ-cover on sn1 . Take N1 (ϵ, F, sn1 ) = ∞ if no finite ϵ-cover exists. Definition C.1.5 (Pseudo-Dimension). A real-valued function class F has pseudo-dimension dimF defined as the VC dimension of the function class 174 induced by the sub-level set of functions of F. In other words, define function class H = {(s, y) 7→ sign(f (s) − y) : f ∈ F}, then dimF = VC-dimension(H). C.2 Generalization Analysis of Fitted Q Evaluation In this section we prove the following statement for Fitted Q Evaluation (FQE). Theorem C.2.1 (Guarantee for FQE - General Case (theorem 5.2.1)). Under Assumption C.1, for ϵ > 0 & δ ∈ (0, 1), after K iterations of Fitted Q Evaluation  4  2 (Algorithm 3), for n = O Cϵ2 (log Kδ + dimF log Cϵ2 + log dimF ) , we have with probability 1 − δ: γ 1/2 q 2γ K/2 C  π Q(π) − QK (π) ≤ β (2d + ϵ) + . µ F (1 − γ)3/2 (1 − γ)1/2 Theorem C.2.2 (Guarantee for FQE - Bellman Realizable Case). Under Assumptions C.1-C.2, for any ϵ > 0, δ ∈ (0, 1), after K iterations of Fit 4 2 ted Q Evaluation (Algorithm 3), when n ≥ 24·214·C log Kδ + dimF log 320C + 2 ϵ ϵ2  log(14e(dimF + 1)) , we have with probability 1 − δ: Q(π) − QK (π) ≤ 2γ K/2 C  γ 1/2 q β ϵ + . µ (1 − γ)3/2 (1 − γ)1/2 We first focus on theorem C.2.2, analyzing FQE assuming a sufficiently rich function class F so that the Bellman evaluation update Tπ is closed wrt F (thus inherent Bellman evaluation error is 0). We call this the Bellman evaluation realizability assumption. This assumption simplifies the presentation of our bounds. After analyzing FQE under this Bellman realizable setting, we will turn to error bound for general, non-realizable setting in theorem C.2.1 (also theorem 5.2.1). The main difference in the non-realizable setting is the appearance of an extra term dπF our final bound. Error bound for single iteration - Bellman realizable case Assumption C.2 (Bellman evaluation realizability). We consider function classes F sufficiently rich so that ∀f, Tπ f ∈ F. We begin with the following result bounding the error for a single iteration of FQE, under “training” distribution µ ∼ πD . 175 Proposition C.2.3 (Error bound for single iteration). Let the functions in F also be bounded by C, and let dimF denote the pseudo-dimension of the function class F. We have with probability at least 1 − δ: ∥Qk − Tπ Qk−1 ∥µ < ϵ, when n ≥ 24·214·C ϵ2 4  2 log 1δ + dimF log 320C + log(14e(dimF + 1)) ϵ2  Remark C.2.4. Note from proposition C.2.3 that the dependence of sample e 1 ), which is better than previously known analysis complexity n here on ϵ is O( ϵ2 for Fitted Value Iteration [150] and FittedPolicyQ (continuous version of Fitted e 1 ). The finite sample analysis of LSTD Q Iteration [11]) dependence of O( ϵ4 1 e [119] showed an O( ϵ2 ) dependence using linear function approximation. Here we prove similar convergence rate for general non-linear (bounded) function approximators. Proof of Proposition C.2.3. Recall the training target in round k is yi = ci + γQk−1 (s′i , π(s′i )) for i = 1, 2, . . . , n, and Qk ∈ F is the solution to the following regression problem: n 1X Qk = arg min (f (si , ai ) − yi )2 . n f ∈F i=1 Consider random variables (s, a) ∼ µ and y = c(s, a) + γQk−1 (s′ , π(s′ )) where s′ ∼ p(·|s, a). By this definition, Tπ Qk−1 is the regression function that minimizes square loss min E|h(s, a) − y|2 out of all functions h (not necessarily X×A h:R 7→R in F). This is due to (Tπ Qk−1 )(s̃, ã) = E [y|s = s̃, a = ã] by definition of the Bellman operator. Consider Qk−1 fixed and we now want to relate the learned function Qk over finite set of n samples with the regression function over the whole data distribution via uniform deviation bound. We use the following lemma: Lemma C.2.5 ([76], theorem 11.4. Original version [124], theorem 3). Consider random vector (X, Y ) and n i.i.d samples (Xi , Yi ). Let m(x) be the (optimal) regression function under square loss m(x) = E[Y |X = x]. Assume |Y | ≤ B a.s. and B ≤ 1. Let F be a set of function f : Rd 7→ R and let |f (x)| ≤ B. 176 Then for each n ≥ 1 n 1X P ∃f ∈ F : E|f (X) − Y | − E|m(X) − Y | − |f (Xi ) − Yi |2 n i=1 ( 2 ! −|m(Xi ) − Yi | 2 2  ≥ ϵ · α + β + E|f (X) − Y | − E|m(X) − Y | 2 2  ) βϵ ϵ2 (1 − ϵ)αn , F, xn1 exp − , 20B 214(1 + ϵ)B 4 ! ≤ 14 sup N1 xn 1 ! where α, β > 0 and 0 < ϵ < 1/2 To apply this lemma, first note that since Tπ Qk−1 is the optimal regression function1 , we have: h Eµ (Qk (s, a) − y)2 i h = Eµ (Qk (s, a) − Tπ Qk−1 (s, a) + Tπ Qk−1 (s, a) − y)2 h i i = Eµ (Qk (s, a) − Tπ Qk−1 (s, a))2 ] + Eµ [(Tπ Qk−1 (s, a) − y)2 . Thus h ∥Qk − Tπ Qk−1 ∥2µ = E (Qk (s, a) − Tπ Qk−1 (s, a))2 h i i h i = E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 , where by definition E (Qk (s, a) − Tπ Qk−1 (s, a))2 = Z (Qk (s, a) − Tπ Qk−1 (s, a))2 µ(dx, da) = Z (Qk (s, a) − Tπ (s, a))2 µx (dx)πD (a|dx) h i f ∼ µ, Next, given a fixed data set D k n o (C.3) P ∥Qk − Tπ Qk−1 ∥2µ > ϵ ( h i h i h i h i ) = P E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 > ϵ ( ≤ P E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 n n 1X 1X (Qk (si , ai ) − yi )2 − (Tπ Qk−1 (si , ai ) − yi )2 > ϵ −2· n i=1 n i=1 ! ) (C.4) 1 It is easy to see that if m(x) = E[y|s] is the regression function then for any function f (x), we have E [(f (x) − m(x))(m(x) − y) = 0] 177 ( h i h = P E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 − i n h i 1X (Qk (si , ai ) − yi )2 − (Tπ Qk−1 (si , ai ) − yi )2 n i=1 h i h i 1 > (ϵ + E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 ) 2 ) ( h i h ≤ P ∃f ∈ F : E (f (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 − (C.5) i n h i 1X (f (si , ai ) − yi )2 − (Tπ Qk−1 (si , ai ) − yi )2 n i=1 h i h i ϵ 1 ϵ ≥ ( + + E (f (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 ) 2 2 2     ϵ nϵ ≤ 14 sup N1 . (C.6) , F, xn1 · exp − 80C 24 · 214C 4 xn 1 ) Equation (C.4) uses the definition of Qk = arg min n1 f ∈F 2 i=1 (f (si , ai ) − yi ) and Pn the fact that Tπ Qk−1 ∈ F, thus making the extra term a positive addition. Equation (C.5) is due to rearranging the terms. Equation (C.6) is an application of lemma C.2.5. We can further bound the empirical covering number by invoking the following lemma due to Haussler [87]: Lemma C.2.6 ([87], Corollary 3). For any set X, any points x1:n ∈ X n , any class F of functions on X taking values in [0, C] with pseudo-dimension dimF < ∞, and any ϵ > 0: N1 (ϵ, F, xn1 ) ≤ e(dimF + 1) 2eC ϵ !dimF . Applying lemma C.2.6 to equation (C.6), we have the inequality 320C 2 ≤ 14·e·(dimF +1) ϵ !dimF nϵ ·exp − . P ∥Qk − T 24 · 214C 4 (C.7)   24·214·C 4 1 320C 2 Thus, when n ≥ log δ + dimF log ϵ2 + log(14e(dimF + 1)) : ϵ2 n π o Qk−1 ∥2µ > ϵ   ∥Qk − Tπ Qk−1 ∥ρ < ϵ, with probability at least 1−δ. Notice that the dependence of sample complexity e 1 ), which is better than previously known analyses for other n here on ϵ is O( ϵ2 approximate dynamic programming algorithms such as Fitted Value Iteration [150], FittedPolicyQ [12, 11] with dependence of O( ϵ14 ). 178 Error bound for single iteration - Bellman non-realizable case We now give similar error bound for the general case, where Assumption C.2 does not hold. Consider the decomposition (C.8) ∥Qk − Tπ Qk−1 ∥2µ h i h = E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 ( h i i h = E (Qk (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 i n n 1X 1X 2 −2· (Qk (si , ai ) − yi ) − (Tπ Qk−1 (si , ai ) − yi )2 n i=1 n i=1 !) n n 1X 1X + 2· (Qk (si , ai ) − yi )2 − (Tπ Qk−1 (si , ai ) − yi )2 n i=1 n i=1 ( !) = component_1 + component_2. Splitting the probability of error into two separate bounds. We saw from the previous section (equation (C.7)) that 640C 2 P(component_1 > ϵ/2) ≤ 14 · e · (dimF + 1) ϵ !dimF nϵ · exp − . 48 · 214C 4 (C.9) π We no longer have component_2 ≤ 0 since T Qk−1 ∈ / F. Let   f ∗ ≡ arg inf ∥f − Tπ Qk−1 ∥2µ . f ∈F 1 Pn Since Qk = arg min n i=1 (f (si , ai ) − yi )2 , we can upper-bound component_2 f ∈F by n n 1X 1X (f ∗ (si , ai ) − yi )2 − (Tπ Qk−1 (si , ai ) − yi )2 component_2 ≤ 2 · n i=1 n i=1 We can treat f ∗ as a fixed function, unlike random function Qk , and use standard concentration inequalities to bound the empirical average from the expectation. Let random variable z = ((s, a), y), zi = ((si , ai ), yi ), i = 1, . . . , n and let ! h(z) = (f ∗ (s, a) − y)2 − (Tπ Qk−1 (s, a) − y)2 . We have |h(z)| ≤ 4C 2 . We will derive a bound for ! n 1X ϵ h(zi ) − Eh(z) > + Eh(z) , P n i=1 4 using Bernstein inequality[145]. First, using the relationship h(z) = (f ∗ (s, a) + Tπ Qk−1 (s, a)−2y)(f ∗ (s, a)−Tπ Qk−1 (s, a)), the variance of h(z) can be bounded 179 by a constant factor of Eh(z), since h Var(h(z)) ≤ Eh(z)2 ≤ 16C 2 E (f ∗ (s, a) − Tπ Qk−1 (s, a))2  h i i h = 16C 2 E (f ∗ (s, a) − y)2 − E (Tπ Qk−1 (s, a) − y)2 i = 16C 2 Eh(z). (C.10) (C.11) Equation (C.10) stems from Tπ Qk−1 being the optimal regression function. Now we can apply equation (C.11) and Bernstein inequality to obtain ! n 1X ϵ P h(zi ) − Eh(z) > + Eh(z) n i=1 4 n 1X ϵ Var(h(z)) ≤P h(zi ) − Eh(z) > + n i=1 4 16C 2  ≤ exp  −  = exp − Thus n  ϵ Var + 16C 2 4 2 2Var + 2 4C3 n  Var ϵ + 16C 2 4 32C 2 + 8C3  2 ϵ Var + 16C 2 4  2  ! ≤ ...     ≤ exp −  n  ϵ Var + 16C 2 4 2 32C 2 + 8C3  2  ϵ Var + 16C 2 4   1 nϵ − . 32 · 128 + 3 C 2 !  ≤ exp n ϵ 1X 3 nϵ h(zi ) − 2Eh(z) > P 2· ≤ exp − · 2 . (C.12) n i=1 2 416 C Now we have " # n n 1X 1X h(zi ) = 2 · h(zi ) − 2Eh(z) + 4Eh(z). component_2 ≤ 2 · n i=1 n i=1 Using again the fact that Tπ Qk−1 is the optimal regression function # " h ! i   h Eh(z) = ED (f ∗ (s, a) − y)2 − ED (Tπ Qk−1 (s, a) − y)2 h = ED (f ∗ (s, a) − Tπ Qk−1 (s, a))2 i i = inf ∥f − Tπ Qk−1 ∥2µ . (C.13) f ∈F Combining equations (C.9), (C.12) and (C.13), we can conclude that n o P ∥Qk − Tπ Qk−1 ∥2µ − 4 inf ∥f − Tπ Qk−1 ∥2µ > ϵ (C.14) f ∈F 640C 2 ≤ 14 · e · (dimF + 1) ϵ implying !dimF n nϵ 3 nϵ · exp − + exp − · 2 , 4 416 C 48 · 214C   o P ∥Qk − Tπ Qk−1 ∥µ − 2 inf ∥f − Tπ Qk−1 ∥µ > ϵ !dimF nϵ2 · exp − 48 · 214C 4  (C.15) f ∈F 640C 2 ≤ 14 · e · (dimF + 1) ϵ2  ! 3 nϵ2 + exp − · . 416 C 2 (C.16) ! 180 We now can further upper-bound the term 2 inf ∥f − Tπ Qk−1 ∥µ ≤ 2 sup inf ∥f − Tπ f ′ ∥µ = 2dπF , f ∈F f ′ ∈F f ∈F (the worst-case inherent Bellman evaluation error), leading to the final bound for the Bellman non-realizable case. One may wish to further remove the inherent Bellman evaluation error from our error bound. However, counter-examples exist where the inherent Bellman error cannot generally be estimated using function approximation (see section 11.6 of [191]). Fortunately, inherent Bellman error can be driven to be small by choosing rich function class F (low bias), at the expense of more samples requirement (higher variance, through higher pseudo-dimension dimF ). While the bound in (C.16) looks more complicated than the Bellman realizable case in equation C.7, note that the convergence rate will still be O( n12 ). Bounding the error across iterations Previous sub-sections C.2 and C.2 have analyzed the error of FQE for a single iteration in Bellman realizable and non-realizable case. We now analyze how errors from different iterations flow through the FQE algorithm. The proof borrows the idea from lemma 3 and 4 of [150] for fitted value iteration (for value function V instead of Q), with appropriate modifications for our off-policy evaluation context. Recall that C π , Qπ denote the true value function and action-value function, respectively, under the evaluation policy π. And CK = E[QK (s, π(s))] denote the value function associated with the returned function QK from algorithm 3. Our goal is to bound the difference C π − CK between the true value function and the estimated value of the returned function QK . Let the unknown state-action distribution induced by the evaluation policy π be ρ. We first bound the loss ∥Qπ − QK ∥ρ under the “test-time” distribution ρ of (s, a), which differs from the state-action µ induced by data-generating policy πD . We will then lift the loss bound from QK to CK . Step 1: Upper-bound the value estimation error Let ϵk−1 = Qk − Tπ Qk−1 ∈ X , C. We have for every k that Qπ − Qk = Tπ Qπ − Tπ Qk−1 + ϵk−1 = γP π (Qπ − Qk−1 ) + ϵk−1 . (Qπ is fixed point of T π ) 181 Thus by simple recursion Qπ − QK = K−1 X γ K−k−1 (P π )K−k−1 ϵk + γ K (P π )K (Qπ − Q0 ) k=0 X (1 − γ)γ K−k−1 1 − γ K+1 K−1 = (P π )K−k−1 ϵk K+1 1−γ 1 − γ k=0 " (1 − γ)γ K π K π + (P ) (Q − Q0 ) 1 − γ K+1 # X 1 − γ K+1 K−1 αk Ak ϵk + αK AK (Qπ − Q0 ) , = 1−γ k=0 where for simplicity of notations, we denote (1 − γ)γ K−k−1 (1 − γ)γ K αk = for k < K, α = K 1 − γ K+1 1 − γ K+1 " # (C.17) Ak = (P π )K−k−1 , AK = (P π )K . Note that Ak ’s are probability kernels and αk ’s are deliberately chosen such P that k αk = 1. We can apply point-wise absolute value on both sides of (C.17) with |f | being the short-hand notation for |f (s, a)| and inequality holds point-wise. By triangle inequalities: " # X 1 − γ K+1 K−1 π π |Q − QK | ≤ αk Ak |ϵk | + αK AK |Q − Q0 | . (C.18) 1−γ k=0 Step 2: Bounding ∥Qπ − QK ∥ρ for any unknown distribution ρ. To handle distribution shift from µ to ρ, we decompose the loss as follows: π ∥Q Z − QK ∥2ρ = ρ(dx, da) (Qπ (s, a) − QK (s, a))2 1 − γ K+1 ≤ 1−γ #2 Z " K−1 X 1 − γ K+1 ≤ 1−γ #2 Z k=0 "K−1 X #2 k=0 "K−1 X " " #2 ! αk Ak |ϵk | + αK AK |Q − Q0 | (s, a) π (from(C.18)) # αk (Ak ϵk ) + αK (AK (Q − Q0 )) 2 ∗ 2 (s, a) (Jensen) 1 − γ K+1 Z αk Ak ϵ2k + αK AK (Q∗ − Q0 )2 (s, a). (Jensen) ≤ 1−γ k=0 Using assumption C.1 (assumption 5.1), we can bound each term ρAk as: " # ρAk = ρ(P π )K−k−1 ≤ µβµ (K − k − 1). (definition C.1.3) Thus ∥Qπ − QK ∥2ρ h i 1−γ K+1 2 1−γ X 1 − γ K−1 ≤ γ K−k−1 βµ (K − k − 1) ∥ϵk ∥2µ + αK (2C)2 . K+1 1−γ k=0 " # Assumption C.1 (stronger than necessary for proof of FQE) can be used to 182 upper-bound the first order concentration coefficient:   X γ  γ (1 − γ) γ m βµ (m) ≤ (1 − γ)2 mγ m−1 βµ (m) = βµ . 1−γ 1−γ m≥0 m≥1 X This gives the upper-bound for ∥Qπ − QK ∥2ρ as " π ∥Q − QK ∥2ρ ≤ 1 − γ K+1 1−γ #2 " γ βµ max ∥ϵk ∥2µ k (1 − γ)(1 − γ K+1 ) # K (1 − γ)γ + (2C)2 K+1 1−γ (C.19) 1 − γ K+1 γ ≤ βµ max ∥ϵk ∥2µ + (1 − γ)γ K (2C)2 2 k (1 − γ) 1 − γ γ γK 2 ≤ (2C)2 . β max ∥ϵ ∥ + µ k µ k (1 − γ)3 1−γ Using a2 + b2 ≤ (a + b)2 for nonnegative a, b, we conclude that ! q γ K/2 γ 1/2 π βµ max ∥ϵk ∥µ + 2C . (C.20) ∥Q − QK ∥ρ ≤ k (1 − γ)3/2 (1 − γ)1/2 Step 3: Desired Bound Now we can choose ρ to be the state-action distribution by the evaluation policy π. The error bound on the value function Q follows simply by integrating inequality (C.20) over state-action pairs induced by π. The final error across iterations can be related to individual iteration error by ! q γ 1/2 γ K/2 π 2C . |Q − QK | ≤ βµ max ∥ϵk ∥µ + (C.21) k (1 − γ)3/2 (1 − γ)1/2 " # Finite-sample guarantees for Fitted Q Evaluation Combining results from (C.7), (C.16) and (C.21), we have the final guarantees for FQE under both realizable and general cases. Realizable Case - Proof of theorem C.2.2. From (C.7), when n ≥ N ∗ ≡   24·214·C 4 K 320C 2 log + dim log + log(14e(dim + 1)) , we have ∥ϵk ∥µ < ϵ with F F 2 2 ϵ δ ϵ probability at least 1 − δ/K for any 0 ≤ k < K. Thus we conclude that for any ϵ > 0, 0 < δ < 1, after K iterations of Fitted Q Evaluation, the value estimate returned by QK satisfies: ! q γ 1/2 γ K/2 π |Q − QK | ≤ βµ ϵ + 2C (1 − γ)3/2 (1 − γ)1/2 holds with probability 1 − δ when n ≥ N ∗ . This concludes the proof of theorem C.2.2. Non-realizable Case - Proof of theorem C.2.1 and theorem 5.2.1. 183 Similarly, from (C.16) we have n o P ∥Qk − Tπ Qk−1 ∥µ − 2 inf ∥f − Tπ Qk−1 ∥µ > ϵ (C.22) f ∈F !dim F 640C 2 3 nϵ2 nϵ2 ≤ 14 · e · (dimF + 1) + exp − · . · exp − ϵ2 416 C 2 48 · 214C 4 Since inf f ∈F ∥f − Tπ Qk−1 ∥µ ≤ suph∈F inf f ∈F ∥f − Tπ h∥µ = dπF (the inherent Bellman evaluation error), similar arguments to the realizable case lead to the conclusion that for any ϵ > 0, 0 < δ < 1, after K iterations of FQE: ! q γ 1/2 γ K/2 π π |Q − QK | ≤ βµ (2dF + ϵ) + 2C (1 − γ)3/2 (1 − γ)1/2 !  4 ! 2  holds with probability 1 − δ when n = O Cϵ2 (log Kδ + dimF log Cϵ2 + log dimF ) , thus finishes the proof of theorem C.2.1. e 1 ) dependency of n is significant improvement Note that in both cases, the O( ϵ2 over previous finite-sample analysis of related approximate dynamic programming algorithms [150, 12, 11]. This dependency matches that of previous analysis using linear function approximators from [118, 119] for LSTD and LSPI algorithms. Here our analysis, using similar assumptions, is applicable for general non-linear, bounded function classes, which is an improvement over convergence rate of O( n14 ) in related approximate dynamic programming algorithms [11, 12, 150]. 184 Appendix D CHAPTER 4 APPENDIX D.1 Equivalence between Regularization and Constraint Satisfaction Formulating Different Regularized Policy Learning Problems as Constrained Policy Learning In this section, we provide connections between regularized policy learning and our constrained formulation (OPT). Although this chapter focuses on batch policy learning, here we are agnostic between online and batch learning settings. Entropy regularized RL. The standard reinforcement learning objective, ∗ either in online or batch setting, is to find a policy πstd that minimizes the ∗ long-term cost (equivalent to maximizing the accumulated rewards): πstd = P arg minπ t E(st ,at )∼π [c(st , at )] = arg minπ E(s,a)∼µπ [c(s, a)]. Maximum entropy reinforcement learning [78] augments the cost with an entropy term, such ∗ that the optimal policy maximizes its entropy at each visited state: πMaxEnt = arg minπ E(s,a)∼µπ [c(s, a)] − λH(π(·|s)). As discuseed by [78], the goal is for the agent to maximize the entropy of the entire trajectory, and not greedily maximizing entropy at the current time step (i.e., Boltzmann exploration). Maximum entropy policy learning was first proposed by [232, 231] in the context of learning from expert demonstrations. Entropy regulazed RL/IL is equivalent to our problem (OPT) by simply set C(π) = E(st ,at )∼π [c(st , at )] (standard RL objective), and g(s, a) = π(a|s) log π(a|s), and thus G(π) = −H(π) ≤ τ. Smooth imitation learning (& Regularized imitation learning). This is a constrained imitation learning problem studied by [122]: learning to mimic smooth behavior in continuous space from human desmonstrations. The data collected from human demonstrations is considered to be fixed and given a priori, thus the imitation learning task is also a batch policy learning problem. The proposed approach from [122] is to view policy learning as a function regularization problem: policy π = (f, g) is a combination of functions f and h, where f belongs to some expressive function class F (e.g., decision trees, neural networks) and h ∈ H with certifiable smoothness property (e.g., linear models). Policy learning is the solution to the functional regularization 185 problem π = arg minf,g Es∼µπ ∥f (s) − πE (s)∥ + λ ∥h(s) − πE (s)∥, where πE is the expert policy. This constrained imitation learning setting is equivalent to our problem (OPT) as follows: C(π) = C((f, h)) = Es∼µπ ∥f (s) − πE (s)∥ and G(π) = G((f, h)) = minh′ ∈H ∥h′ (s) − πE (s)∥ ≤ τ . Regularizing RL with expert demonstrations / Learning from imperfect demonstrations. Efficient exploration in RL is a well-known challenge. Expert demonstrations provide a way around online exploration to reduce the sample complexity for learning. However, the label budget for expert demonstrations may be limited, resulting in a sparse coverage of the state space compared to what the online RL agent can explore [91]. Additionally, expert demonstrations may be imperfect [157]. Some recent work proposed to regularize standard RL objective with some deviation measure between the learning policy and (sparse) expert data [91, 157, 88]. For clarity we focus on the regularized RL objective for Q-learning in [91], which is defined as J(π) = JDQ (Q) + λ1 Jn (Q) + λ2 JE (Q) + λ3 JL2 (Q), where JDQ (Q) is the standard deep Q-learning loss, Jn (Q) is the n-step return loss, JE (Q) is the imitation learning loss given by JE (Q) = maxa∈A [Q(s, a) + ℓ(aE , a) − Q(s, aE )], and JL2 (Q) is an L2 regularization loss applied to the Q-network to prevent overfitting to a small expert dataset. The regularization parameters λ’s are obtained by hyperparameter tuning. This approach provides a bridge between RL and IL, whose objective functions are fundamentally different (see AggreVate by [175] for an alternative approach). We can cast this problem into (OPT) as C(π) = CDQ (Q) + λ3 CL2 (Q) (standard RL objective), and two constraints: g1 (π) = Es∼µπ [maxa∈A Q(s, a) + ℓ(aE , a) − Q(s, aE )], and g2 (s, a) = Es∼µπ [ct + γct+1 + . . . + γ n−1 ct+n−1 + min′a γ n Q(st+n , a′ ) − Q(st , a)]. Here g1 captures the loss w.r.t. expert demonstrations and g2 reflects the n-step return constraint. More generally, one can define the imitation learning constraint as G(π) = Es∼µπ ℓ(π(s), πE (s)) for an appropriate divergence definition between π(s) and πE (s) (defined at states where expert demonstrations are available). Conservative policy improvement. Many policy search algorithms perform small policy update steps, requiring the new policy π to stay within a neighborhood of the most recent policy iterate πk to ensure learning stability [125, 179, 146, 3]. This simply corresponds to the definition of G(π) = distance(π, πk ) ≤ 186 τ , where distance is typically KL-divergence or total variation distance between the distribution induced by π and πk . For KL-divergence, the single k (a|s) timestep cost g(s, a) = −π(a|s) log( ππ(a|s) ). Equivalence of Regularization and Constraint Viewpoint - Proof of Proposition 6.2.1 Regularization =⇒ Constraint: Let λ > 0 and π ∗ be optimal policy in Regularization. Set τ = G(π ∗ ). Suppose that π ∗ is not optimal in Constraint. Then ∃π ∈ Π such that G(π) ≤ τ and C(π) < C(π ∗ ). We then have C(π) + λ⊤ G(π) < C(π ∗ ) + λ⊤ τ = C(π ∗ ) + λ⊤ G(π ∗ ), which contradicts the optimality of π ∗ for Regularization problem. Thus π ∗ is also the optimal solution of the Constraint problem. Constraint =⇒ Regularization: Given τ and let π ∗ be the corresponding optimal solution of the Constraint problem. The Lagrangian of Constraint is given by L(π, λ) = C(π) + λ⊤ G(π), λ ≥ 0. Then π ∗ = arg min max L(π, λ). π∈Π Let λ≥0 λ∗ = arg max min L(π, λ). λ≥0 π∈Π Slater’s condition implies strong duality. By strong duality and the strong maxmin property [26], we can exchange the order of maximization and minimization. Thus π ∗ is the optimal solution of min π∈Π ∗ ⊤ C(π) + (λ∗ )⊤ (G(π) − τ ). Removing the constaint (λ ) τ , we have that π ∗ is the optimal solution of the Regularization problem with λ = λ∗ . And since π ∗ = ̸ arg min C(π), we must have λ∗ ≥ 0. π∈Π 187 D.2 Convergence Proofs Convergence of Meta-algorithm - Proof of Proposition 6.3.1 Let us evaluate the empirical primal-dual gap of the Lagrangian after T iterations: 1X max L(πbT , λ) = max L(πt , λ) (D.1) λ λ T t o(T ) 1X L(πt , λt ) + (D.2) ≤ T t T 1X o(T ) ≤ L(π, λt ) + ∀π ∈ Π (D.3) T t T b ) + o(T ) ∀π. = L(π, λ (D.4) T T b and linearity Equations (D.1) and (D.4) are due to the definition of πbT and λ T of L(π, λ) wrt λ and the distribution over policies in Π. Equation (D.2) is due to the no-regret property of Online-algorithm. Equation (D.3) is true since πt is best response wrt λt . Since equation (D.4) holds for all π, ) we can conclude that for T sufficiently large such that o(T ≤ ω, we have T b maxλ L(πbT , λ) ≤ minπ L(π, λT ) + ω , which will terminate the algorithm. b ) ≥ min L(π, λ b ). Algo Note that we always have maxλ L(πbT , λ) ≥ L(πbT , λ T π T 4’s convergence rate depends on the regret bound of the Online-algorithm √ procedure. Multiple algorithms exist with regret scaling as Ω( T ) (e.g., online gradient descent with regularizer, variants of online mirror descent). In that case, the algorithm will terminate after O( ω12 ) iterations. Empirical Convergence Analysis of Main Algorithm - Proof of Theorem 6.4.1 By choosing normalized exponentiated gradient as the online learning subroutine, we have the following regret bound after T iterations of the main algorithm 5 (Chapter 2 of [182]) for any λ ∈ Rm+1 , ∥λ∥1 = B: + T T 1X 1X b b L(π L(π t , λ) ≤ t , λt ) + T t=1 T t=1 B log(m+1) + ηG2 BT η T . (D.5) 188 B log(m+1) +ηG2 BT η b Denote ωT = to simplify notations. By the linearity of L(π, λ) T in both π and λ, we have for any λ that T T eqn (D.5) 1X linearity 1 X b b b b L(πT , λ) = L(πt , λ) ≤ L(π t , λt ) + ωT T t=1 T t=1 best response πt ≤ linearity = T 1X b π b T , λt ) + ωT L( T t=1 b )+ω . b π bT , λ L( T T b )+ω . b π b π b T , λ) ≤ L( bT , λ Since this is true for any λ, maxλ L( T T On the other hand, for any policy π, we also have T T best response πt 1 X 1X b ) linearity b b b L(π, λ = L(π, λt ) ≥ L(π T t , λt ) T t=1 T t=1 eqn (D.5) ≥ linearity = T 1X b b L(π t , λT ) − ω T T t=1 b )−ω . b π bT , λ L( T T b ) ≥ L( b ) − ω , leading to b b π bT , λ Thus minπ L(π, λ T T T b ) ≤ L( b ) + ω − (L( b ) − ω ) = 2ω b π b b π b π b T , λ) − min L(π, bT , λ bT , λ max L( λ T T T T T T λ π After T iterations of the main algorithm 5, therefore, the empirical primal-dual gap is bounded by 2 B log(m+1) + 2ηG2 BT η b b b b max L(πT , λ) − min L(π, λT ) ≤ . π λ T In particular, if we want the gap to fall below a desired threshold ω, setting the online learning rate η = 4Gω2 B will ensure that the algorithm converges after 16B 2 G2 log(m+1) iterations. ω2 D.3 End-to-end Generalization Analysis of Main Algorithm In this section, we prove the following full statement of theorem 6.4.3. Note that to lessen notation, we define V = C + BG to be the bound of value functions under considerations in algorithm 5. Theorem D.3.1 (Generalization bound of algorithm 5). Let π ∗ be the optimal policy to problem OPT. Let K be the number of iterations of FQE and FQI. Let πb be the policy returned by our main algorithm 5, with termination threshold  4 2 ω. For any ϵ > 0, δ ∈ (0, 1), when n ≥ 24·214·V log K(m+1) + dimF log 320V + 2 ϵ δ ϵ2  log(14e(dimF + 1)) , we have with probability at least 1 − δ: C(πb ) ≤ C(π ∗ ) + ω +  (4 + B)γ q K/2 C ϵ + 2γ V ρ (1 − γ)3 189 and G(πb ) ≤ τ + 2 V +ω γ 1/2 q 2γ K/2 V  + C ϵ + . ρ B (1 − γ)3/2 (1 − γ)1/2 Let πb = T1 t πt be the returned policy T iterations, with corresponding dual b= 1 P λ. variable λ t t T P By the stopping condition, the empirical duality gap is less than some threshb ≤ ω where L(π, b π b b b b , λ) − min L(π, old ω, i.e., max L( λ) λ) = C(π) + m+1 λ∈R+ π∈Π ,∥λ∥1 =B b λ (G(π) − τ ). We first show that the returned policy approximately satisfies the constraints. The proof of theorem D.3.1 will make use of the following empirical constraint satisfaction bound: ⊤ Lemma D.3.2 (Empirical constraint satisfactions). Assume that the conb straints G(π) ≤ τ are feasible. Then the returned policy πb approximately satisfies all constraints C +ω . max (gbi (πb ) − τi ) ≤ 2 i=1:m+1 B Proof. We consider max (gbi (πb ) − τi ) > 0 (otherwise the lemma statement is i=1:m+1 trivially true). The termination condition implies that b − b π b , λ) L( max λ∈Rm+1 ,∥λ∥1 =B + b ⊤ (G( b π b ) − τb) ≥ =⇒ λ max b π b , λ) ≥ −ω L( λ∈Rm+1 ,∥λ∥1 =B + b π b ) − τb) − ω. λ⊤ (G( (D.6) We relax the RHS of Eq (D.6) by setting λ[j] = B for j = arg max [gbi (πb ) − τi ] i=1:m+1 and λ[i] = 0 ∀i ̸= j yields: b ⊤ (G( b π b ) − τ ). B max [gbi (πb ) − τi ] − ω ≤ λ i=1:m+1 (D.7) b Given π such that G(π) ≤ τ , also by the termination condition: b − L(π, b ≤ b π b b , λ) L( λ) max λ∈Rm+1 ,∥λ∥1 =B + b ≤ ω. b π b b , λ) − min L(π, L( λ) π∈Π Thus implies b ≤ L(π, b + ω = C(π) b ⊤ (G(π) b π b b b b b , λ) L( λ) +λ − τ ) ≤ C(π) + ω. (D.8) Combining what we have from equation (D.8) and (D.7): b ⊤ (G( b − C( b π b π b π b b π b ) − τb) = L( b , λ) b ) ≤ C(π) b ). B max [gbi (πb ) − τbi ] − ω ≤ λ + ω − C( i=1:m+1 b b π b ) ≤ −C completes the proof. Rearranging and bounding C(π) ≤ C and C( 190 We now return to the proof of theorem D.3.1, our task is to lift empirical error to generalization bound for main objective and constraints. Denote by ϵF QE the (generalization) error introduced by the Fitted Q Evaluation procedure (algorithm 3) and ϵF QI the (generalization) error introduced by the Fitted Q Iteration procedure (algorithm 13). For now we keep ϵF QE and ϵF QI unspecified (to be specified shortly). That is, for each t = 1, 2, . . . , T , we have with probability at least 1 − δ: ∗ ⊤ ∗ C(πt ) + λ⊤ t (G(πt ) − τ ) ≤ C(π ) + λt (G(π ) − τ ) + ϵF QI . Since π ∗ satisfies the constraints, i.e., G(π ∗ ) − τ ≤ 0 componentwise, and λt ≥ 0, we also have with probability 1 − δ ∗ L(πt , λt ) = C(πt ) + λ⊤ t (G(πt ) − τ ) ≤ C(π ) + ϵF QI . (D.9) Similarly, with probability 1 − δ, all of the following inequalities are true b b C(π t ) + ϵF QE ≥ C(πt ) ≥ C(πt ) − ϵF QE (D.10) b b G(π t ) + ϵF QE 1 ≥ G(πt ) ≥ G(πt ) − ϵF QE 1 (row wise for all m constraints). (D.11) Thus with probability at least 1 − δ: ⊤ b ⊤ b L(πt , λt ) = C(πt ) + λ⊤ t (G(πt ) − τ ) ≥ C(πt ) + λt (G(πt ) − τ ) − ϵF QE (1 + λt 1) ⊤ b b ≥ C(π t ) + λt (G(πt ) − τ ) − ϵF QE (1 + B) b = L(π t , λt ) − ϵF QE (1 + B). (D.12) Recall that the execution of mixture policy πb is done by uniformly sampling one policy πt from {π1 , . . . , πT }, and rolling-out with πt . Thus from equations ∗ b (D.9) and (D.12), we have Et∼U [1:T ] L(π t , λt ) ≤ C(π ) + ϵF QI + (1 + B)ϵF QE w.p. 1 − δ. In other words, with probability 1 − δ: T 1X ∗ b L(π t , λt ) ≤ C(π ) + ϵF QI + (1 + B)ϵF QE . T t=1 Due to the no-regret property of our online algorithm (EG in this case): T 1X b b b , λ) − ω = C( b π b π b ) + max λ⊤ (G( b) − τ ) − ω L(π t , λt ) ≥ max L(π λ λ T t=1 b π b ) − τ ≤ 0 componentwise, choose λ[i] = 0, i = 1, 2, . . . , m and λ[m + 1] = If G( B. Otherwise, we can choose λ[j] = B for j = arg max [gbi (πb ) − τ [i]] and i=1:m+1 λ[i] = 0 ∀i ̸= j. We can see that max m+1 λ∈R+ ,∥λ∥1 =B b π b ) − τ ) ≥ 0. Therefore: λ⊤ (G( b π b ) − ω ≤ C(π ∗ ) + ϵF QI + (1 + B)ϵF QE with probability at least 1 − δ. C( 191 Combined with the first term from equation (D.10): C(πb ) − ϵF QE − ω ≤ C(π ∗ ) + ϵF QI + (1 + B)ϵF QE or C(πb ) ≤ C(π ∗ ) + ω + ϵF QI + (2 + B)ϵF QE . (D.13) We now bring in the generalization error results from our standalone analysis of FQI (appendix D.5) and FQE (Chapter 5 appendix C.2) into equation (D.13). Specifically, when ! 24 · 214 · V 4 K(m + 1) 320V 2 n≥ log + dimF log + log(14e(dimF + 1)) , ϵ2 δ ϵ2 when FQI and FQE are run with K iterations, we have the guarantee that for any ϵ > 0, with probability at least 1 − δ q  2γ K/2 C ϵ + 2γ V C(πb ) ≤ C(π ∗ ) + ω + µ (1 − γ)3 | {z FQI generalization error } ! γ 1/2 (2 + B) q γ K/2 + Cµ ϵ + 2V (1 − γ)3/2 (1 − γ)1/2 | {z (2+B)× FQE generalization error ≤ C(π ∗ ) + ω +  (4 + B)γ q K/2 C ϵ + 2γ V . µ (1 − γ)3 } (D.14) b π b ) ≤ τ + 2 C+ω From lemma D.3.2, G( ≤ τ + 2 V B+ω . From equation (D.11), for B b each t=1,2,. . . ,T, we have G(π t ) ≥ G(πt ) − ϵF QE 1 with probability 1 − δ. Thus P  T  X b b b b b = πt )P(π b = πt ) G(π ) ≥ G(π ) − ϵF QE 1 = P(G(π t ) ≥ G(πt ) − ϵF QE 1|π t=1 1 = 1 − δ. T Therefore, we have the following generalization guarantee for the approximate satisfaction of all constraints: ! q V +ω γ 1/2 γ K/2 G(πb ) ≤ τ + 2 + Cµ ϵ + 2V . (D.15) B (1 − γ)3/2 (1 − γ)1/2 Inequalities (D.14) and (D.15) complete the proof of theorem D.3.1 (and theorem 6.4.3). ≥ T (1 − δ) D.4 Preliminaries to Analysis of Fitted Q Iteration (FQI) This section follows the same setup and structure as Chapter 5 Appendix section C.1. We show improved convergence rate from O(n−4 ) to O(n−2 ), where n is the number of samples in data set D. 192 To be consistent with the notation of this chapter, we use the convention C(π) as the value function that denotes long-term accumulated cost, instead of using V (π) denoting long-term rewards in the traditional RL literature. Our notation for Q function is similar to the RL literature - the only difference is that the optimal policy minimizes Q(x, a) instead of maximizing. We denote the bound on the value function as C (alternatively if the single timestep cost c is bounded by c, then C = 1−γ ). For simplicity, the standalone analysis of FQI concerns only with the cost objective c. Dealing with cost c + λ⊤ g offers no extra difficulty - in that case we simply augment the bound of the value function to V = C + BG. We begin by recalling a few ideas and definitions from Chapter 5 Appendix section C.1. Reminder from 5 Appendix section C.1 Recall, the Bellman optimality operator T : B(X ; C) 7→ B(X ; C) as: (TQ)(s, a) = c(s, a) + γ Z min Q(s′ , a′ )p(ds′ |s, a). (D.16) S a′ ∈A The optimal value functions are defined as usual by C ∗ (s) = sup C π (s) and π Q∗ (s, a) = sup Qπ (s, a) ∀(s, a) ∈ X . π Denote the state-action data generating distribution as µ, induced by some data-generating (behavior) policy πD , that is, (si , ai ) ∼ µ for (si , ai , s′i , ci ) ∈ D. Note that data set D is formed by multiple trajectories generated by πD . For each (si , ai ), we have s′i ∼ p(·|si , ai ) and ci = c(si , ai ). For any (measurable) function f : X 7→ R, define the µ-weighted ℓ2 norm of f as ∥f ∥2µ = R R 2 2 X f (s, a) µ(ds, da) = X f (s, a) µs (ds)πD (a|ds). Similarly for any other stateR action distribution ρ, ∥f ∥2ρ = X f (s, a)2 ρ(ds, da) Definition D.4.1 (Inherent Bellman Error). Given a function class F and a chosen distribution ρ, the inherent Bellman error of F is defined as: dF = d(F, TF) = sup inf ∥f − Th∥ρ , h∈F f ∈F where ∥·∥ρ is the ρ−weighted ℓ2 norm and T is the Bellman optimality operator defined in (D.16) Definition D.4.2 (Concentration coefficient of state-action distribution). Given data generating distribution µ ∼ πD , initial state distribution d0 . For 193 m ≥ 0, and an arbitrary sequence of stationary policies {πm }m≥1 let d(d0 P π1 P π2 . . . P πm ) . βµ (m) = sup dµ π1 ,...,πm ∞ (βµ (m) = ∞ if the future state distribution d0 P π1 P π2 . . . P πm is not absolutely continuous w.r.t. µ, i.e, d0 P π1 P π2 . . . P πm (s, a) > 0 for some µ(s, a) = 0) Assumption D.1. βµ = (1 − γ)2 P m≥1 mγ m−1 βµ (m) < ∞. Definition D.4.3 (Random L1 Norm Covers). Let ϵ > 0, let F be a set of functions S 7→ R, let sn1 = (s1 , . . . , sn ) be n fixed points in S. Then a collection of functions Fϵ = {f1 , . . . , fN } is an ϵ-cover of F on sn1 if n n 1X 1X ∀f ∈ F, ∃f ′ ∈ Fϵ : | f (si ) − f ′ (si )| ≤ ϵ. n i=1 n i=1 The empirical covering number, denote by N1 (ϵ, F, sn1 ), is the size of the smallest ϵ-cover on sn1 . Take N1 (ϵ, F, sn1 ) = ∞ if no finite ϵ-cover exists. Definition D.4.4 (Pseudo-Dimension). A real-valued function class F has pseudo-dimension dimF defined as the VC dimension of the function class induced by the sub-level set of functions of F. In other words, define function class H = {(x, y) 7→ sign(f (x) − y : f ∈ F}, then dimF = VC-dimension(H). D.5 Finite-Sample Analysis of Fitted Q Iteration (FQI) Algorithm and Discussion Algorithm 13 Fitted Q Iteration with Function Approximation: FQI(c) [56] Require: Collected data set D = {si , ai , s′i , ci }ni=1 . Function class F 1: Initialize Q0 ∈ F randomly 2: for k = 1, 2, . . . , K do 3: Compute target yi = ci + γ mina Qk−1 (s′i , a) ∀i e = {(si , ai ), yi }n 4: Build training set D k i=1 5: Solve a supervised learning problem: P Qk = arg min n1 ni=1 (f (si , ai ) − yi )2 f ∈F Ensure: πK (·) = arg min QK (·, a) (greedy policy with respect to the returned a function QK ) The analysis of FQI (algorithm 13) follows analogously from the analysis of FQE (Chapter 5 Appendix C.2). For brevity, we skip certain detailed derivations, especially those that are largely identical to FQE’s analysis. 194 To the best of our knowledge, a finite-sample analysis of FQI with general non-linear function approximation has not been published (Continuous FQI from [11] is in fact a Fitted Policy Iteration algorithm and is different from algo 13). In principle, one can adapt existing analysis of fitted value iteration [150] and FittedPolicyQ [12, 11] to show that under similar assumptions, among policies greedy w.r.t. functions in F, FQI will find ϵ− optimal policy using e 1 ) samples. We derive an improved analysis of FQI with general nonn = O( ϵ4 e 1 ). linear function approximations, with better sample complexity of n = O( ϵ2 We note that the appendix of [120] contains an analysis of LinearFQI showing similar rate to ours, albeit with linear function approximators. In this section, we prove the following statement: Theorem D.5.1 (Guarantee for FQI - General Case (theorem 6.4.2)). Under Assumption D.1, for any ϵ > 0, δ ∈ (0, 1), after K iterations of Fitted Q  4  2 Iteration (algorithm 13), for n = O Cϵ2 (log Kδ + dimF log Cϵ2 + log dimF ) , we have with probability 1 − δ:  2γ q K/2 C ∗ − C(πK ) ≤ β (2d + ϵ) + 2γ C , µ F (1 − γ)3 where πK is the policy greedy with respect to the returned function QK , and C ∗ is the value of the optimal policy. The key steps to the proof follow similar scheme to the proof of FQE. We first bound the error for each iteration, and then analyze how the errors flow through the algorithm. Single iteration error bound ∥Qk − TQk−1 ∥µ Here µ is the state-action distribution induced by the data-generating policy πD . We begin with the decomposition: h i h ∥Qk − TQk−1 ∥2µ = E (Qk (s, a) − y)2 − E (TQk−1 (s, a) − y)2 ( h i h = E (Qk (s, a) − y)2 − E (TQk−1 (s, a) − y)2 i i n n 1X 1X −2· (Qk (si , ai ) − yi )2 − (TQk−1 (si , ai ) − yi )2 n i=1 n i=1 (D.17) !) n n 1X 1X 2 + 2· (Qk (si , ai ) − yi ) − (TQk−1 (si , ai ) − yi )2 n i=1 n i=1 ( = component_1 + component_2. !) 195 For T the Bellman (optimality) operator (equation D.16), TQk−1 is the regression function that minimizes square loss min E|h(s, a) − y|2 , with the X h:R 7→R random variables (s, a) ∼ µ and y = c(s, a) + γ mina′ Qk−1 (s′ , a′ ) where s′ ∼ p(s′ |s, a). Invoking lemma C.2.5 and following the steps similar to equations (C.4),(C.5),(C.6), and (C.7) from Chapter 5 Appendix C.2, we can bound the first component as 640C 2 P(component_1 > ϵ/2) ≤ 14 · e · (dimF + 1) ϵ !dimF nϵ · exp − . 48 · 214C 4 (D.18) P 2 n 1 Let f ∗ = arg inf ∥f − TQk−1 ∥µ . Since Qk = arg min n i=1 (f (si , ai ) − yi )2 , we f ∈F   f ∈F can upper-bound component_2 by ! n n 1X 1X ∗ 2 2 component_2 ≤ 2 · (f (si , ai ) − yi ) − (TQk−1 (si , ai ) − yi ) . n i=1 n i=1 Let random variable z = ((s, a), y), zi = ((si , ai ), yi ), i = 1, . . . , n and let h(z) = (f ∗ (s, a) − y)2 − (TQk−1 (s, a) − y)2 . We have |h(z)| ≤ 4C 2 . We can derive a bound for ! n 1X ϵ P h(zi ) − Eh(z) > + Eh(z) , n i=1 4 using Bernstein inequality, similar to equations (C.10) and (C.11) from Chapter 5 Appendix C.2 to obtain: " # !   n 1X ϵ 3 nϵ P 2· h(zi ) − 2Eh(z) > ≤ exp − · . (D.19) n i=1 2 416 C 2 Now we have " # n n 1X 1X h(zi ) = 2 · h(zi ) − 2Eh(z) + 4Eh(z). component_2 ≤ 2 · n i=1 n i=1 Since h i h Eh(z) = EDek (f ∗ (s, a) − y)2 − EDek (TQk−1 (s, a) − y)2 h = EDek (f ∗ (s, a) − TQk−1 (s, a))2 i i = inf ∥f − TQk−1 ∥2µ . (D.20) f ∈F Combining equations (D.18), (D.19) and (D.20), we obtain that n o P ∥Qk − TQk−1 ∥2µ − 4 inf ∥f − TQk−1 ∥2µ > ϵ f ∈F !dimF 640C 2 ≤ 14 · e · (dimF + 1) ϵ   3 nϵ · . + exp − 416 C 2 nϵ · exp − 48 · 214C 4   (D.21) 196 Propagation of error bound for ∥Q∗ − QπK ∥ρ The analysis of error propagation for FQI is more involved than that of FQE, but the proof largely follows the error propagation analysis in lemma 3 and 4 of [150] in the fitted value iteration context (for V function). We include the Q function’s (slighly more complicated) derivation here for completeness. Recall that πK is greedy wrt the learned function QK returned by FQI. We aim to bound the difference C ∗ − C πK between the optimal value function and that πK . For a (to-be-specified) distribution ρ of state-action pairs (different from the data distribution µ), we bound the generalization loss ∥Q∗ − QπK ∥ρ Step 1: Upper-bound the propagation error (value). Let ϵk−1 = Qk − TQk−1 . We have that ∗ ∗ ∗ ∗ ∗ Q∗ − Qk = Tπ Q∗ − Tπ Qk−1 + Tπ Qk−1 − TQk−1 + ϵk−1 ∗ ≤ Tπ Q∗ − Tπ Qk−1 + ϵk−1 (b/c TQk−1 ≥ Tπ Qk−1 ) ∗ = γP π (Q∗ − Qk−1 ) + ϵk−1 . Thus by recursion Q∗ −QK ≤ PK−1 K−k−1 π∗ K−k−1 ∗ γ (P ) ϵ +γ K (P π )K (Q∗ −Q ) k k=0 0 Step 2: Lower-bound the propagation error (value). Similarly Q∗ − Qk = TQ∗ − Tπk−1 Q∗ + Tπk−1 Q∗ − TQk−1 + ϵk−1 ≥ Tπk−1 Q∗ − TQk−1 + ϵk−1 (as TQ∗ ≥ Tπk−1 Q∗ ) ≥ Tπk−1 Q∗ − Tπk−1 Qk−1 + ϵk−1 (b/c πk−1 greedy wrt Qk−1 ) = γP πk−1 (Q∗ − Qk−1 ) + ϵk−1 . And by recursion: ∗ Q − QK ≥ K−1 X γ K−k−1 (P πK−1 P πK−2 . . . P πk+1 )ϵk k=0 + γ K (P πK−1 P πK−2 . . . P π0 )(Q∗ − Q0 ). Step 3: Upper-bound the propagation error (policy). Beginning with a decomposition of value wrt to policy πK : ∗ ∗ ∗ Q∗ − QπK = Tπ Q∗ − Tπ QK + Tπ QK − TπK QK + TπK QK − TπK QπK (a) ∗ ∗ ≤ (Tπ Q∗ − Tπ QK ) + (TπK QK − TπK QπK ) ∗ = γP π (Q∗ − QK ) + γP πK (QK − QπK ) ∗ = γP π (Q∗ − QK ) + γP πK (QK − Q∗ + Q∗ − QπK ), ∗ where (a) uses Tπ QK ≤ TQK = TπK QK . This leads to (I − γP πK )(Q∗ − ∗ QπK ) ≤ γ(P π − P πK )(Q∗ − QK ) The operator (I − γP πK ) is invertible and 197 (I − γP πK )−1 = P m≥0 γ m (P πK )m is monotonic. Thus Q∗ − QπK ∗ ≤ γ(I − γP πK )−1 (P π − P πK )(Q∗ − QK ) ∗ = γ(I − γP πK )−1 P π (Q∗ − QK ) − γ(I − γP πK )−1 P πK (Q∗ − QK ). (D.22) Applying inequalities from Step 1 and Step 2 to the RHS of (D.22), we have ∗ πK Q −Q ≤ (I − γP ) πK −1 " K−1 X  ∗  γ K−k (P π )K−k − P πK P πK−1 . . . P πk+1 ϵk k=0  π∗ #  + γ K+1 (P )K+1 − (P πK P πK−1 . . . P π0 ) (Q∗ − Q0 ) . (D.23) Next we apply point-wise absolute value on RHS of (D.23), with |ϵk | being the short-hand notation for |ϵk (s, a)| point-wise. Using triangle inequalities and rewriting (D.23) in a more compact form ([150]): X 2γ(1 − γ K+1 ) K−1 ≤ αk Ak |ϵk | + αK AK |Q∗ − Q0 | , 2 (1 − γ) k=0 " ∗ πK Q −Q # K−k−1 K where αk = (1−γ)γ for k < K, αK = (1−γ)γ and 1−γ K+1 1−γ K+1 i h 1−γ ∗ Ak = (I − γP πK )−1 (P π )K−k + P πK P πK−1 . . . P πk+1 for k < K 2 i h 1−γ ∗ AK = (I − γP πK )−1 (P π )K+1 + P πK P πK−1 . . . P π0 . 2 Note that Ak ’s are probability kernels that combine the P πi terms and αk ’s P are chosen such that k αk = 1. Step 4: Bounding ∥Q∗ − QπK ∥2ρ for any test distribution ρ. This step handles distribution shift from µ to ρ (similar to Step 2 from subsection C.2 of Chapter 5 Appendix C.2). Using Jensen twice: #2 X 2γ(1 − γ K+1 ) Z K−1 αk Ak ϵ2k + αK AK (Q∗ − Q0 )2 (s, a). (1 − γ)2 k=0 Using assumption D.1 (assumption 6.1), each term ρAk is bounded as h i 1−γ ∗ ρAk = ρ(I − γP πK )−1 (P π )K−k + P πK P πK−1 . . . P πk+1 2 h i 1−γ X m ∗ γ ρ(P πK )m (P π )K−k + P πK P πK−1 . . . P πk+1 = 2 m≥0 " " # ∥Q∗ − QπK ∥2ρ ≤ ≤ (1 − γ) X m≥0 γ m βµ (m + K − k)µ. (def D.4.2) 198 Thus ∥Q∗ − QπK ∥2ρ 2γ(1 − γ K+1 ) ≤ (1 − γ)2 " #2 " 1 × 1 − γ K+1 K−1 X # (1 − γ)2 X γ m+K−k−1 βµ (m + K − k) ∥ϵk ∥2µ + αK (2C)2 m≥0 k=0 K+1 #2 " 1 (1 − γ)γ K 2 β max ∥ϵ ∥ + (2C)2 µ k µ k 1 − γ K+1 1 − γ K+1 2γ(1 − γ K+1 ) ≤ (1 − γ)2 #2 " # 2γ(1 − γ ) ≤ (1 − γ)2 (a) " " #2  1 γK 2 β max ∥ϵ ∥ + (2C)2 µ k µ k 1 − γ K+1 1 − γ K+1 # 2γ βµ max ∥ϵk ∥2µ + γ K (2C)2 , ≤ k (1 − γ)2 where (a) uses (Assump. D.1). Using a2 + b2 ≤ (a + b)2 for nonnegative a, b, we thus conclude that q  2γ K/2 ∥Q∗ − QπK ∥ρ ≤ β max ∥ϵ ∥ + 2γ C . (D.24) µ k µ k (1 − γ)2 Step 5: Bounding C ∗ − C πK Using the performance difference lemma (lemma 1 Es∼dπK A∗ [s, a]. We can upper6.1 of [100], which states that C ∗ − C πK = − 1−γ "  a∼πK bound the performance difference of value function as C ∗ − C πK (D.25) 1 1 = Es∼dπK [C ∗ (s) − Q∗ (s, a)] = Es ∼dπK [C ∗ (s) − Q∗ (s, πK (s))] 1 − γ a∼πK 1−γ (a) 1 ≤ Es ∼dπK [Q∗ (s, π ∗ (s)) − QK (s, π ∗ (s)) 1−γ + QK (s, πK (s)) − Q∗ (s, πK (s))] ≤ 1 Es ∼dπK |Q∗ (s, π ∗ (s)) − QK (s, π ∗ (s))| 1−γ + |QK (s, πK (s) − Q∗ (s, πK (s))|  1  ∗ ∥Q − QπK ∥dπ ×π∗ + ∥Q∗ − QπK ∥dπ ×πK K K 1−γ q  2γ ≤ βµ max ∥ϵk ∥µ + 2γ K/2 C , (D.26) k (1 − γ)3 where (a) is greedy, and (b) is a 2-norm upper bound of the 1-norm. Note ∗ that inequality (D.26) follows from (D.24) by specifying ρ = d0 P πK P π and ρ = d0 P πK P πK , respectively (d0 is the initial state distribution). (b) ≤ 199 Finite-sample guarantees for Fitted Q Iteration From (D.21) we have: n o P ∥Qk − TQk−1 ∥µ − 2 inf ∥f − TQk−1 ∥µ > ϵ f ∈F !dim F nϵ2 640C 2 ≤ 14 · e · (dimF + 1) · exp − ϵ2 48 · 214C 4 ! 3 nϵ2 . · + exp − 416 C 2 Note that inf f ∈F ∥f − TQk−1 ∥µ ≤ suph∈F inf f ∈F ∥f − Th∥µ = dF (the inherent Bellman error from equation D.16). Combining with equation (D.26), we have the conclusion that for any ϵ > 0, 0 < δ < 1, after K iterations of Fitted Q Iteration, and for πK the greedy policy wrt QK : q  2γ K/2 π β (2d + ϵ) + 2γ C C ∗ − CK ≤ µ F (1 − γ)3  4 ! 2  holds with probability 1 − δ when n = O Cϵ2 (log Kδ + dimF log Cϵ2 + log dimF ) . Note that compared to the Fitted Value Iteration analysis of [150], our error includes an extra factor 2 for dF . Statement for the Bellman-realizable Case To facilitate the end-to-end generalization analysis of theorem 6.4.3, we include a version of FQI analysis under Bellman-realizable assumption in this section. The theorem is a consequence of previous analysis in this section. Assumption D.2 (Bellman evaluation realizability). We consider function classes F sufficiently rich so that ∀f, Tf ∈ F. Theorem D.5.2 (Guarantee for FQI - Bellman-realizable Case). Under Assumption D.1 and D.2, for any ϵ > 0, δ ∈ (0, 1), after K iterations of Fitted   4 2 Q Iteration, for n ≥ 24·214·C log Kδ + dimF log 320C + log(14e(dimF + 1)) , we ϵ2 ϵ2 have with probability 1 − δ:  2γ q K/2 C ∗ − C(πK ) ≤ β ϵ + 2γ C , µ (1 − γ)3 where πK is the policy greedy with respect to the returned function QK , and C ∗ is the value of the optimal policy. D.6 Additional Instantiation of Meta-Algorithm (algorithm 4) We provide an additional instantiation of the meta-algorithm described in this chapter, with Online Gradient Descent (OGD) [234] and Least-Squares Policy Iteration (LSPI) [115] as subroutines. Using LSPI requires a feature map φ 200 such that any state-action pair can be represented by k features. The value function is linear in parameters represented by φ. Policy representation is simplified to a weight vector w ∈ Rk . Similar to our main algorithm 5, OGD updates require bounded parameters λ. We thus introduce hyper-parameter B as the bound of λ in ℓ2 norm. The gradient update is projected to the ℓ2 ball when the norm of λ exceeds B (line 15 of algorithm 14). Algorithm 14 Batch Learning under Constraints using Online Gradient Descent and Least-Squares Policy Iteration Require: Dataset D = {xi , ai , x′i , ci , gi }ni=1 ∼ πD . Online algorithm parameters: ℓ2 norm bound B, learning rate η Require: Number of basis function k. Basis function φ (feature map for state-action pairs) 1: Initialize λ1 = (0, . . . , 0) ∈ Rm 2: for each round t do 3: Learn wt ← LSPI(c + λ⊤ // LSPI with cost c + λ⊤ t g) t g b 4: Evaluate C(wt ) ← LSTDQ(wt , c) // Algo 16 with πt , cost c b 5: Evaluate G(w ) ← LSTDQ(w , g) // Algo 16 with πt , cost g t t Pt 1 6: wbt ← t t′ =1 wt′ P b w b b b ) ← 1 Pt′ G(w b bt ) ← 1t tt′ =1 C(w 7: C( t′ ), G(w t t′ ) t =1 t P t 1 b 8: λt ← t t′ =1 λt′ b ⊤ g) b⊤g 9: Learn we ← LSPI(c + λ // LSPI with cost c + λ t t b w) b w) e ← LSTDQ(w, e c), G( e ← LSTDQ(w, e g) 10: Evaluate C(   b b w b w bt ) + λ⊤ (G( bt ) − τ ) 11: L C( max = max λ,∥λ∥2 ≤B 12: 13: 14: b b ⊤ (G( b e +λ b w) e − τ) L min = C(w) t b b if Lmax − Lmin ≤ ω then   Return πbt greedy w.r.t wbt i.e., πbt (x) = arg mina∈A wbt⊤ φ(s, a) ∀s 15: λ b λt+1 = P(λt − η(G(π t ) − τ )) where projection P(λ) = B max{B,∥λ∥ } 2 Algorithm 15 Least-Squares Policy Iteration: LSPI(c) [115] Require: Stopping criterion ϵ 1: Initialize w ′ ← w0 2: repeat 3: w ← w′ 4: w′ ← LSTDQ(w, c) 5: until ∥w − w ′ ∥ ≤ ϵ   Ensure: Policy weight w i.e., π(s) = arg mina∈A w⊤ φ(s, a) ∀s 201 Algorithm 16 LSTDQ(w, c) [115] f←0 1: Initialize A e 2: Initialize b ← 0 3: 4: // k × k matrix // k × 1 vector for each (s, a, s′ , c) ∈ D do e) a′ = arg minea∈A w⊤ φ(x′ , a e ←A e + φ(s, a) φ(s, a) − γφ(x′ , a′ ) A e 6: b ← eb + φ(s, a)c e −1e e←A b 7: w e Ensure: w 5: D.7
⊤ Additional Experimental Details Environment Descriptions and Procedures Figure D.1: Depicting the FrozenLake and CarRacing environments. Frozen Lake. The environment is a 8x8 grid as seen in Figure D.1 (left), based on OpenAi’s FrozenLake-v0. In each episode, the agent starts from S and traverse to goal G. While traversing the grid, the agent must avoid the pre-determined holes denoted by H. If the agent steps off of the grid, the agent returns to the same grid location. The episode terminates when the agent reaches the goal or falls into a hole. The arrows in Figure D.1 (left) is an example policy returned by our algorithm, showing an optimal route. Denote Sholes as the set of all holes in the grid and Sgoal = {sgoal } is a singleton set representing the goal in the grid. The contrained batch policy learning problem is: min C(π) = E[I(s′ ̸∈ Sgoals )] = P(s′ ̸∈ {sgoal }) s.t. G(π) = E[I(s ∈ Sholes )] = P(s ∈ Sholes ) ≤ τ. π∈Π ′ ′ (D.27) We collect 5000 trajectories by selecting an action randomly with probability .95 and an action from a DDQN-trained model with probability .05.Furthermore we set B = 30 and η = 50, the hyperparameters of our Exponentiated Gradient subroutine. We set the threshold for the constraint τ = .1. Car Racing. The environment is a racetrack as seen in Figure D.1 (right), 202 modified from OpenAi’s CarRacing-v0. In each state, given by the raw pixels, the agent has 12 actions: a ∈ A = {(i, j, k)|i ∈ {−1, 0, 1}, j ∈ {0, 1}, k ∈ {0, .2}}. The action tuple (i, j, k) cooresponds to steering angle, amount of gas applied and amount of brake applied, respectively. In each episode, the agent starts at the same point on the track and must traverse over 95% of the track, given by a discretization of 281 tiles. The agent recieves a reward of + 1000 for each unique tile over which the agent drives. The agent 281 receives a penalty of −.1 per-time step. Our collected dataset takes the form: D = {(st−6 , st−3 , st ), at , (st−3 , st , st+3 ), ct , g0,t , g1,t } where si denotes the image at timestep i and at is applied 3 times between st and st+3 . This frame-stacking option is common practice in online RL for Atari and video games. In our collected dataset D, the maximum horizon is 469 time steps. The first constraint concerns accumulated number of brakes, a proxy for smooth driving or acceleration. The second constraint concerns how far the agent travels away from the center of the track, given by the Euclidean distance between the agent and the closest point on the center of the track. Let Nt be the number of tiles that is collected by the agent in time t. The constrained batch policy learning problem is: ∞ X 1000 Nt + .1)] min E[ γ t (− π∈Π 281 t=0 s.t. G0 (π) = E[ G1 (π) = E[ ∞ X γ t I(at ∈ Abraking )] ≤ τ0 (D.28) t=0 ∞ X t γ d(ut , vt )] ≤ τ1 . t=0 We instatiate our subroutines, FQE and FQI, with multi-layered CNNs. Furthermore we set B = 10 and η = .01, the hyperparameters of our Exponentiated Gradient subroutine. We set the threshold for the constraint to be about 75% of the value exhibited by online RL agent trained by DDQN [211]. Additional Discussion for the Car Racing Experiment Regularized policy learning and grid-search. We perform grid search over a range of regularization parameters λ for both Least-Squares Policy Iteration LSPI ([115]) and Fitted Q Iteration - FQI ([56]). The results, seen from the the first and second plot of Figure D.2, show that one-shot regularized learning has difficulty learning a policy that satisfies both constraints. We augment LSPI with non-linear feature mapping from one of our best performing FQI model 203 Figure D.2: (First and Second figures) Result of 2-D grid-search for one-shot, regularized policy learning for LSPI (left) and FQI (right). (Third and Fourth figures) value range of individual policies in our mixtured policy and data generating policy πD for main objective (left) and cost constraint (right). (using CNNs representation). While both regularized LSPI and regularized FQI can achieve low main objective cost, the constraint cost values tend to be sensitive with the λ step. Overall for the whole grid search, about 10% of regularized policies satisfy both constraints, while none of the regularized LSPI policy satisfies both constraints. Mixture policy and de-randomization. As our algorithm returned a mixture policy, it is natural to analyze the performance of individual policies in the mixture. The third and fourth plot from Figure D.2 show the range of performance of individual policy in our mixture (purple band). We compare individual policy return with the stochastic behavior of the data generation policy. Note that our policies satisfy constraints almost always, while the individual policy returned in the mixture also tends to outperform πD with respect to the main objective cost. Off-policy evaluation standalone comparison. Typically, inverse propensity scoring based methods call for stochastic behavior and evaluation policies [164, 195]. However in this domain, the evaluation policy and environment are both deterministic, with long horizon (the max horizon is D is 469). Consequently Per-Decision Importance Sampling typically evaluates the policy as 0. In general, off-policy policy evaluation in long-horizon domains is known to be challenging [134, 75]. We augment PDIS by approximating the evaluation policy with a stochastic policy, using a softmin temperature parameter. However, 204 PDIS still largely shows significant errors. For Doubly Robust and Weighted Doubly Robust methods, we train a model of the environment as follows: • a 32-dimensional representation of state input is learned using variational autoencoder. Dimensionality reduction is necessary to aid accuracy, as original state dimension is 96 × 96 × 3. • an LSTM is used to learn the transition dynamics P (z(s′ )|z(s), a), where z(s) is the low-dimensional representation learned from previous step. Technically, using a recurrent neural networks is an augmentation to the dynamical modeling, as true MDPs typically do not require long-term memory. • the model is trained separately on a different dataset, collected indendently from the dataset D used for evaluation. The architecture of our dynamics model is inspired by recent work in modelbased online policy learning [77]. However, despite our best effort, learning the dynamics model accurately proves highly challenging, as the horizon and dimensionality of this domain are much larger than popular benchmarks in the OPE literature [98, 202, 61]. The dynamics model has difficulty predicting the future state several time steps away. Thus we find that the long-horizon, model-based estimation component of DR and WDR in this high-dimensional setting is not sufficiently accurate. For future work, a thorough benchmarking of off-policy evaluation methods in high-dimensional domains would be a valuable contribution. 205 Appendix E CHAPTER 5 APPENDIX E.1 Notation and Overview Acronym Term LTL MDP, LDBA AMEC MEC/EC LCP FindAMEC PR PT, NB-PT VI AP Σ S A AA P Pπ C X τ η-greedy(π) φ P [π |= φ] Π, Πmax β D b P P̃ P E n(s, a) ψ(n) ψ −1 (ρ) ϵV ϵφ ϵPR ϵPT ϵL PR ϵL PT α Lα PR Linear Temporal Logic Markov Decision Process. Limit Determinisitic Buchi Automaton Accepting MEC Maximal EC, End Component LTL Constrained Planning, Algo 6 Subroutine to assist in finding AMECs PlanRecurrent. Subroutine to plan in AMECs, Algo 7 PlanTransient/ NoBlockPlanTransient. Subroutine to plan to AMECs, Algo 8/ Algo 19 Value Iteration Subroutine, Algo 17 Atomic Proposition Alphabet Σ = 2AP State Space Action Space. A(s) allowable actions in state s. Restricted Action Space. AA (s) ⊆ A(s) allowable actions in state s ∈ A ⊆ S. Transition Function Markov Chain induced by π in P Cost Function Product-MDP Run or trajectory in an MDP = (1 − η)π + ηUnif(AA ) with 0 ≤ η ≤ 1, defined with respect to A ⊆ S LTL Specification/Formula/Task Probability that a policy satisfies the task Class of stochastic policies, π ∈ Π with maximal P [π |= φ] Lower bound on minimum, nonzero transition probability Dataset tracking all tuples (s, a, s′ ) simulated Empirical estimate of P from data in D Optimistic dynamics returned by VI Plausible transition functions consistent with all the information gathered in D High probability event Number of samples accumulated in (s, a). Also denoted n b(s, a, s′ ) − P (s, a, s′ )| Error bound on maxs′ ∈S |P b(s, a, s′ ) − P (s, a, s′ )| < ρ Number of samples n(s, a) necessary to achieve maxs′ ∈S |P Cost-optimality tolerance wrt. main problem (1) Prob-optimality tolerance wrt. main problem (1) V error input into PR, ϵPR = ϵ7λ 2ϵV error input into PT, ϵPT = 9 2ϵPR Convergence condition for VI in PR, ϵL PR = 3 c ϵ ϵ L Convergence condition for VI in PT, ϵPT = min4V̄PT φ Aperiodicity Coefficient α ∈ (0, 1). Pα,π = αPπ + (1
− α)I T Lα PR v(s) = mina∈AA (s) C(s, a) + α minp∈P(s,a) p v + (1 − α)v(s) ∀s ∈ A  LPT dPR dPT VT λ Jπ gπ V̄ ∆(M ) ¯A ∆ i 
min mina∈AA (s) C(s, a) + minp∈P(s,a) pT v , V̄ , s ∈ S \ ∪ki=1 Ai LPT v(s) = λgi , s ∈ Ai Convergence operator for PR, dPR (v ′ , v) = maxs∈A (v ′ (s) − v(s)) − mins∈A (v ′ (s) − v(s)) Convergence operator for PT, dPT (vn+1 , vn ) = ∥vn+1 − vn ∥1 Terminal costs. VT = 0 by default Tradeoff between gπ and Jπ Transient cost, conditioned on runs satisfying φ Gain, Average-cost, conditioned on runs satisfying φ Upper bound on Jπ for any π ∈ Π P Coefficient of Ergodicity of matrix M , ∆(M ) = 21 maxij |Mik − Mjk | k Worst-case coefficient of ergodicty in Ai 206 Overview There is a lot of notation that we will be using to get through the analysis. It is important to distinguish the following: Table E.1: Policies and Probabilities. Acronym Term π∗ π π Ai π̃Ai πi pπ , p∗ π∗ pπ i , pi Optimal policy w.r.t (LTL-OPT) Policy returned by LCP (Algo 6) Gain and Prob-optimal policy in AMEC (Ai , AAi ) in dynamics P Gain and Prob-optimal policy in AMEC (Ai , AAi ) in dynamics P̃ A policy in states Ai of an AMEC (Ai , AAi ), ignoring what π does outside of Ai ∗ P [π |= φ] and P [π ∗ |= φ]. Also denoted pπ and pπ Pk Pk ∗ P [π reaches Ai ] , P [π ∗ reaches Ai ] ≥ 0 denoted pi , p∗i (resp), p = p, p = p∗ i=1 i i=1 i Table E.2: Gains. Term Description. (Subscript i or Ai denotes “in AMEC (Ai , AAi )”) P̃ b gπ̃ A i P gπ i P gπ A i Approximated gain of (greedy) optimal policy π̃Ai under optimistic dynamics P̃ Actual gain of policy πi (η-greedy version of policy from PR) under true dynamics P Gain of optimal policy πAi under dynamics P The relationship between these gains is subtle. PlanRecurrent (Algo 7) returns gbπ̃P̃A as the estimate for how good the best greedy policy will be in AMEC i (Ai , AAi ) under dynamics P̃ . But we don’t use the greedy policy, we use the η-greedy policy πi . With πAi being the true gain-optimal policy in dynamics P (in AMEC (Ai , AAi )), then we will find the following relations: gbπ̃P̃A P̃ i | {z } Output from PR ≈ gπ̃A |{z} i ϵL PR 2 ≈ |{z} Lem E.2.7 gπP̃i ≈ |{z} Lem E.2.8 gπPi ≈ |{z} gπPA . i Prop E.2.5 | {z } Actual In general gains g are functions of state: g(s). However, it is well known [166, 66] that in communicating MDPs (each state is reachable from one another by some policy) that the gain of the optimal policy (even if determinstic) is constant – independent of state. Since AMECs are communicating MDPs, then πAi , π̃Ai induce constant gains in P, P̃ respectively. Lastly, the stochastic policy πi makes both P̃ and P recurrent, and so the gain is also constant. We will therefore only be considering the absolute difference between gains rather than L∞ norms (as they coincide). When superscripts are dropped in V , the dynamics are the true dynamics P of the product-MDP X . Once again, the relationships between these value functions is subtle. PlanTransient (Algo 8) returns v as the estimate for how 207 Table E.3: Value Functions. Acronym Term Vπ v ṼπP̃ ṼπP P = J + λg Main objective, value function Vπ,λ π π Approximated value of policy π from PT (Algo 7) in dynamics P̃ Actual value of policy π from PT in dynamics P̃ Actual value of policy π from PT in dynamics P , Pk pi P̃ also denoted Ṽπ = p(Jπ + b g ) + (1 − p) ϵV̄ i=1 p π̃A i φ good π (the greedy policy wrt v) will be in reaching AMECs {(Ai , AAi )}ki=1 , but is optimistic. v is an approximation to ṼπP̃ . Roughly speaking, we will find that they are all similar/related: v |{z} ≈ |{z} ṼπP̃ Ouput from PT Lem E.2.16 ≈ |{z} ṼπP Lem E.2.15 ≈ |{z} Ṽ P∗ π |{z} , Prop E.2.12/E.4.1 An intermediate Value Func. where the last approximation has two different propositions: the first allows the simplifying assumption made in this chapter regarding paths between AMECS, and the second removes that assumption at the expense of increased computation. Finally, ∥ṼπP − ṼπP∗ ∥ ∥VπP − VπP∗ ∥, ≈ |{z} Thm 7.4.1 which involves swapping gbπP̃A in Ṽ i for gπPA . i (E.1) 208 E.2 Analysis: Statements with Proof Sample Complexity Guarantee The number of samples necessary to guarantee an (ϵV , ϵφ , δ)-PAC approximation to the cost-optimal and probability-optimal policy relies factors: β (lower bound on the mininum non-zero transition probability of P ), {cmin , cmax } (bounds on ¯ A (worst-case coefficient of ergodicity for EC (Ai , AA )), the cost function C), ∆ i i V̄ (upper bound on the value function), and λ (tradeoff factor). Recall that an event E captures the scenario where the empirical transition function Pb is close to the true transition function P . E holds with probability at least 1 − δ, see Lem E.2.1. Theorem 7.4.1 (Sample Complexity). Under the event E, Assumption 7.1 and 7.2, after !! k X 1 1 |S|2 V̄ 4 |Ai |2 c2max 2 +λ n = Õ + ¯ 2 β ϵ2V c2min ϵ4φ i=1 (1 − ∆Ai ) samples1 are collected from each state-action pair, the policy π returned by Algorithm 6 is, with probability 1 − δ, simultaneously ϵV -cost optimal and ϵφ -probability optimal, satisfying: (i) |P [π |= φ] − P [π ∗ |= φ] | ≤ ϵφ (ii) ∥Vπ − Vπ∗ ∥∞ < ϵV . (7.5) Comparison To RL Literature. Before presenting the proofs, we briefly compare this guarantee with standard guarantees in model-based reinforcement learning under a generative model. It is important to note that while we show that our guarantee is a sum of 3 terms, a tighter bound would be a max over the 3 terms. To the best of our knowledge, the current state-of-the-art RL 1 (with generative model) guarantee is Õ( (1−γ) 3 ϵ2 ) [6], per state-action pair. Here, 1 H = 1−γ represents the effective horizon in discounted settings. In other words, cmax H is the bound on (their) ∥V ∥. In our case, for the SSP reduction, the ∥∞ effective horizon is H = ∥cṼmin , as this is the expected goal-reaching time in the worst-case (since we do not have any discounting). We estimate ∥Ṽ ∥∞ with upper bound ϵV̄φ . Suppose we set ϵ = min(ϵV , ϵφ ). Focusing just on the center 2 4 term, we have guarantee taking the form, roughly, |S|ϵ2H . Here, the |S|2 comes from a loose upper bound maxs∈S,a∈A ∥Pb (s, a, ·)∥1 = |S|. In fact, as noted in [198], when the MDP is not too chaotic maxs∈S,a∈A ∥Pb (s, a, ·)∥1 = O(1). 1 log(2δ) The lower bound relating to β from [131] is Ω( log(1−β) ) whereas ours is Õ( β1 ). We conjecture that Ω̃( β1 ) samples are required. See Appendix Section E.3. 209 Further, by using careful variance-aware arguments from [198] we can decrease the dependency from H 4 to H 3 . Hence, the SSP guarantee (our center term) and the standard RL guarantee are very similar. The first term β1 does not appear in standard RL literature because there is no constraint verification needed, but in practice will be dominated by the other terms. The last term cmax is also similar to the center term. 1− ¯ A can also be seen as an effective ∆ i horizon, accumulating cmax cost until the accepting component sufficiently mixes. Here, |Ai |2 ≤ |S|2 and, again, comes from the loose upper bound maxs∈Ai ,a∈AAi ∥Pb (s, a, ·)∥1 = |Ai |. Proof of Theorem 7.4.1. We begin by examining the interaction of π ∗ with P . The Markov chain Pπ∗ has a number, say m, of recurrent classes R1 , . . . , Rm , sets of states that are trapping and visited infinitely often once reached. Some of the recurrent classes Ri contain an accepting state s ∈ S ∗ , making any trajectory entering Ri an accepting run, without loss of generality call these R1 , . . . , Rm′ (we just relabel them). Let Aπi∗ (s) = {a ∈ A|πi∗ (s|a) > 0} denote the support of actions taken by πi∗ in state s ∈ Ri . Let Aπi∗ = {Aπi∗ (s)}s∈Ri be ′ the indexed action set in Ri . Then, by definition, {(Ri , Aπi∗ )}m i=1 are accepting EC. By definition, each accepting EC (Ri , Aπi∗ ) must be contained within (or is itself) some AMEC (Ai , Ai ). Fix some accepting EC (Rj , Aπj∗ ). We claim, without loss of generality, (Rj , Aπj∗ ) = (Ai , AAi ) for some index i ∈ 1, . . . , k. To show this, let πAi be the gain optimal, and probability-optimal policy in AMEC (Ai , AAi ): πAi is defined over all states s ∈ Ai and actions a ∈ AAi . Further, consider the modified optimal policy π̃ ∗ (s, a) =   πA (s, a), s ∈ Ai  π ∗ (s, a), otherwise i . Because πAi is prob-optimal (ie. P [π̃ ∗ |= φ|s0 ∈ Ai ] = 1) in Ai then the probability P [π̃ ∗ |= φ] ≥ P [π ∗ |= φ]. Further, Jπ̃∗ ≤ Jπ because any τ that formerly passed through Rj \ Ai now accumulates less cost. Further, gπAi ≤ gπi∗ by definition of optimality in AMEC (Ai , AAi ). Thus, Vπ̃∗ ≤ Vπ∗ . Of course, by definition of optimality, the opposite signs hold: Vπ̃∗ ≥ Vπ∗ and P [π̃ ∗ |= φ] ≤ P [π ∗ |= φ]. Therefore π̃ ∗ and π ∗ are indistinguishable. Repeating the above argument for each (Rj , Aπj∗ ) means the accepting EC of π ∗ are AMECS and, by definition, form some subset of all of the AMECs 210 {(Ai , AAi }ki=1 . In other words, all accepting runs of π ∗ reach states ∪ki=1 Ai . P p∗ Furthermore, gπ∗ = ki=1 pi gπPA where p∗i ≥ 0 is the probability that π ∗ reaches i P Ai and ki=1 p∗i = p. Property (i) now follows as a direct consequence of Prop 7.3.3 and Prop 7.3.2. Recall by Prop 7.3.3 that h i h i |P π reaches ∪ki=1 Ai − ′max P π ′ reaches ∪ki=1 Ai | ≤ ϵφ . π ∈Πmax Prop 7.3.2 implies that once a run enters some Ai , the run is accepted. Remaining runs cannot be accepted since they do not reach any AMEC, the only h i way to be accepted. Hence P [π |= φ] = P π reaches ∪ki=1 Ai . Since we just showed that all accepting runs of π ∗ reach some (Ai , Ai ) then: 0 ≤ P [π ∗ |= φ] − P [π |= φ] h i h ≤ P π ∗ reaches ∪ki=1 Ai − P π reaches ∪ki=1 Ai i ≤ ϵφ . To show Property (ii), first let us define pπi as the probability of π reaching P P AMEC (Ai , AAi ) and, by property (i), ki=1 pπi = ki=1 p∗i = p. The value function given by the Bellman operator LPT (Table 7.1) in Algorithm 8 takes the form k X V̄ pπi P̃ (E.2) gbπ̃A + (1 − p) , Ṽπ (s) = p(Jπ (s) + λ i ϵφ i=1 p where gbπ̃P̃A are the approximated gains for end component (Ai , AAi ) from i Algorithm 7. To see this, there is probability p that π |= φ and achieves P (conditional) expected cost Jπ (s) + λ ki=1 ppi gbπ̃P̃A and prob 1 − p that π ̸|= φ i where all cooresponding trajectories get stuck and accumulate ϵV̄φ cost. Let π̃ now represent the optimal solution to the value function Ṽπ (Algo 8). Therefore we claim: 0 ≤ Vπ − Vπ∗ = Vπ − Ṽπ + Ṽπ − Ṽπ̃ + Ṽπ̃ − Ṽπ∗ + Ṽπ∗ − Vπ∗ + (1 − p) ≤ |Vπ − Ṽπ + (1 − p) | ≤ {z (a) V̄ V̄ − (1 − p) ϵφ ϵφ V̄ V̄ | + |Ṽπ − Ṽπ̃ | + Ṽπ̃ − Ṽπ∗ + |Ṽπ∗ − Vπ∗ − (1 − p) | | {z } | {z } ϵφ ϵφ } (b) (c) | {z (d) } ϵV ϵV ϵV + +0+ ≤ ϵV . 3 3 3 For (a), first we note that gπ = pπ P i i=1 p gπi , by definition of conditional expec- Pk 211 V tation. Let ϵPR = ϵ7λ . Hence, |Vπ − Ṽπ + (1 − p) | {z (a) k k X X pπi P̃ V̄ pπi P | = |p(Jπ (s) + λ gπi ) − p(Jπ (s) + λ gbπ̃A )| i ϵφ i=1 p i=1 p } ≤ λ max |gπPi − gbπ̃P̃A | i=1,...,k i 4ϵPR , Corollary E.2.6 ≤λ 3 ϵV ≤ . 3 By similar argument, for (d), together with earlier argument that gπ∗ = Pk p∗i P i=1 p gπA then we also have that: i k k X X V̄ pπi P p∗i P̃ |Ṽπ∗ − Vπ∗ − (1 − p) | = |p(Jπ∗ (s) + λ gbπ̃A ) − p(Jπ∗ (s) + λ gπA )| i i ϵφ i=1 p i=1 p | {z (d) } ≤ λ max |gπPA − gbπ̃P̃A | i=1,...,k i i ≤ λ max |gπPA − gπPi | + |gπPi − gbπ̃P̃A | i=1,...,k 7ϵPR ≤λ , 3 ϵV ≤ . 3 i i Prop E.2.5 and Corollary E.2.6 Further, we have (c) ≤ 0 holds because π̃ is optimal in Ṽ (either by assuming ∪ki=1 Ai is the correct choice of AMECS, or using Algo 19 instead of planTransient). In either case, (b) ≤ 3ϵ2PT ≤ ϵ3V by Prop E.2.12 or Prop E.4.1, where ϵPT is set to ϵPT = 2ϵ9V , completing the approximation guarantee. We now compute the number of samples, per state-action pair, required by Algorithm 6. By Prop E.2.4, we need n = Õ( β1 ) to verify the support of P . After calculating the AMECs {(Ai , AAi )}ki=1 , we calculate the gain-optimal i |cmax 2 policy πi for each AMEC. By Prop 7.3.2, we need n = Õ(( ϵPR|A(1− ¯ )) ) = ∆ Ai i |cmax Õ(( ϵλ|A )2 ) for each state-action pair in each end component (Ai , AAi ), ¯ V (1−∆Ai ) V since ϵPR = ϵ7λ . Finally, for the transient policy π0 , the SSP reduction requires k |S\∪i=1 Ai |V̄ 2 2 |S\∪k Ai |V̄ 2 2 n = Õ(( ϵPT ϵ2 cmin ) ) = Õ(( ϵV i=1 ) ) for each state-action pair outside of ϵ2φ cmin φ 2ϵV the AMECs, by Prop 7.3.3, since ϵPT = 9 . A similar sample complexity is guarV̄ 2 anteed by using Algo 19 in place of PlanTransient, where n = Õ(( ϵV|S| )2 ) ϵ2 cmin φ |S\∪k Ai |V̄ 2 2 is required in place of n = Õ(( ϵV i=1 ) ). Adding these together yields the ϵ2φ cmin worst-case number of samples necessary in any state-action pair (s, a) ∈ S × A. These sample guarantees hold only when the event E holds, which itself holds 212 with probability 1 − δ (see Lem E.2.1). Corollary 7.4.2 (Gain (Average Cost) Optimality). There exists λ∗ > 0 s.t. for λ > λ∗ , the policy π returned by Alg. 6 satisfies (7.5), gπ = arg minπ′ ∈Πmax gπ′ , and is probability and gain optimal. Proof of Corollary 7.4.2. Fix some λ > 0. Let π ′ = arg minπ∈Πmax gπ . Suppose gπ′ < gπ but Vπ,λ < Vπ′ ,λ , elementwise. In other words, π is the preferred policy. Then, 0 ≤ Vπ′ ,λ − Vπ,λ = Jπ′ + λgπ′ − Jπ − λgπ ≤ max Jπ̃ + λ (gπ′ − gπ ), π̃∈Π | {z <0 maxπ̃∈Π Jπ̃ gπ −gπ′ } since Jπ̃ ≥ 0 for each π̃ ∈ Π. If λ > then we contradict Vπ,λ < Vπ′ ,λ . In particular, if π ′ is the gain optimal policy then for any λ > λ∗ = maxπ̃∈Π Jπ̃ then π ′ is preferred to any other policy π ∈ Π. min{π∈Π|g ̸=g } gπ −g ′ π π′ π 213 High Probability Event and Sample Requirement Definition 7.3.2 (High Probability Event). A high probability event E: E = {∀s, a, s′ ∈ S × A × S, ∀n(s, a) > 1 : |(P (s,a, s′ ) − Pb (s, a, s′ ))| ≤ ψsas′ (n) ≤ ψ(n)}, q where ψsas′ (n) ≡ 2P̂ (s, a, s′ )(1 − P̂ (s, a, s′ )))ξ(n) + 73 ξ(n), ψ(n) ≡ 2 2 |A| 7 ξ(n), and ξ(n) ≡ log( 4n |S| )/(n − 1). 3 δ q 1 ξ(n) + 2 Lemma E.2.1 (High Probability Event holds). The event E holds with probability at least 1 − δ. Proof of Lemma E.2.1 . We start with the anytime version of Theorem 4 of [141] given by Lemma 27 of [199]:  P ∀ n ≥ 1, E[Z] −  1 Zi > n i=1 n X v u u 2V̂ log(4n2 /δ) t n n−1 +  7 log(4n /δ)   ≤ δ, 3(n − 1) 2 for any Zi ∈ [0, 1] iid. By re-setting δ ← |S|2δ|A| , applying union bound over all (s, a, s′ ) ∈ S × A × S, and observing that Zi ∼ P (s, a, s′ ) is a Bernoulli random variable with empirical variance V̂n = Pb (s, a, s′ )(1 − Pb (s, a, s′ )) yields the result: {∀s, a, s′ ∈ S×A×S, ∀n > 1 : |P (s, a, s′ )−Pb (s, a, s′ )| ≤ ψsas′ (n)} w. prob 1−δ Observing that ψsas′ (n) ≤ ψ(n) for all n > 1 because ψsas′ (n) takes on a maximum when Pb (s, a, s′ ) = 12 , completes the proof. Lemma E.2.2 (Inverting E). Fix (s, a, s′ ) ∈ S × A × S. Under the event E, the number of samples ψ −1 (ρ) required to achieve |P (s, a, s′ ) − Pb (s, a, s′ )| ≤ ψsas′ (n) ≤ ψ(n) < ρ is given by: 2 16|S|2 |A| 1 ψ −1 (ρ) = ⌈ 2 log( )⌉ + 3 = Õ( ), ζ ζ 4δ ρ2 where ζ ≡ 3 − √ + 7 2 q 3 ( √ )2 + 12 ρ 7 7 2 2 . Proof of E.2.2. We have ψsas′ < ψ(n) = √x2 + 73 x2 ≤ ρ where x2 = ξ(n) = log(4n2 |S|2 |A|δ −1 ) . Solving the quadratic inequality, we have n−1 q − 7√3 2 + ( 7√3 2 )2 + 12 ρ 7 x≤ 2 ≡ ζ. 214 Hence, we have log(4n2 |S|2 |A|δ −1 ) ≤ ζ2 n−1 log(4n2 |S|2 |A|δ −1 ) =⇒ n ≥ +1 ζ2 1 2 = 2 log(eζ 4n2 |S|2 |A|δ −1 ) ζ q ζ2 2 = 2 log(e 2 4|S|2 |A|δ −1 n). (⋆) ζ | {z } |{z} c1 c2 By Lemma E.2.3, if n > 2c1 log(c1 c2 ) then n > (⋆). Simplifying, 2 16|S|2 |A| n ≥ 2 log( ) + 2. ζ ζ 4δ 2 |A| Selecting ψ −1 (ρ) = ⌈ ζ22 log( 16|S| )⌉ + 3 and noting that ζ = Õ(ρ) completes ζ4δ 2 the proof: n = Õ(1/ρ ). Lemma E.2.3. (Lemma 10 of [105]) If log(c1 c2 ) ≥ 1 and c1 , c2 > 0 then N > 2c1 log(c1 c2 ) =⇒ N > c1 log(c2 N ). 215 FindAMEC proofs Proposition E.2.4. (Support Verification FindAMEC) Under the event E and 2 |A| Assumption 7.1, if n = φFindAMEC (β) = β5 log( 100|S| ) = Õ( β1 ) samples are 2 β δ collected for each state-action pair (s, a) ∈ S × A then the support of P is verified:    0,    Pb (s, a, s′ ) = 0    ∈ [β, 1 − β], otherwise P (s, a, s′ ) = 1, Pb (s, a, s′ ) = 1 . Proof of Prop E.2.4 . Fix (s, a, s′ ) ∈ S × A × S. Suppose Pb (s, a, s′ ) ∈ {0, 1} then by E we have 7 log(4n2 |S|2 |A|/δ) ≤ β. (E.3) 3(n − 1) , we have that if Following the second half of the proof of E.2.2 with ζ 2 = 3β 7 100|S 2 A| 784|S 2 A| 14 5 we take n = φFindAMEC (β) = β log( β 2 δ ) > 3β log( 9β 2 δ ) then we have |P (s, a, s′ ) − Pb (s, a, s′ )| < β. (E.4) Case Pb (s, a, s′ ) = 1. Suppose Pb (s, a, s′ ) = 1. By Eq (E.4), P (s, a, s′ ) > 1 − β. P By Assumption 7.1 together with the fact that x∈S P (s, a, x) = 1 then P (s, a, x) = 0 for any x ̸= s′ . Therefore, P (s, a, s′ ) = Pb (s, a, s′ ) = 1. Case Pb (s, a, s′ ) = 0. Suppose Pb (s, a, s′ ) = 0. By Eq (E.4), P (s, a, s′ ) < β. Hence P (s, a, s′ ) = Pb (s, a, s′ ) = 0, otherwise violating Assumption 7.1. Case, Otherwise. If P (s, a, s′ ) > 1 − β or P (s, a, s′ ) < β then by following the above arguments we’d yield similar contradictions with Assumption 7.1. Hence, P (s, a, s′ ) ∈ [β, 1 − β]. 216 PlanRecurrent proofs Proposition 7.3.2 (PR Convergence & Correctness, Informal). Let πA be the gain-optimal policy in AMEC (A, A). Algorithm 7 terminates after at most   2 c2 max max log2 ϵPR6|A|c repeats, and collects at most n = Õ( ϵ2|A| ¯ A) ¯ A )2 ) samples for (1−∆ (1− ∆ PR ′ each (s, a) ∈ (A, AA ). The η-greedy policy π w.r.t. v (Alg. 7, Line 5) is gain optimal and probability optimal: |gπ − gπA | < ϵPR , P [π |= φ|s0 ∈ A] = 1. We formalize Prop 7.3.2 as follows by adding the necessary PAC statements: Proposition E.2.5 (PR Convergence & Correctness, Formal). Let πA be the gain-optimal policy in AMEC (A, A). Algorithm 7 terminates after at most   |A|2 c2max max log2 ϵPR6|A|c repeats, and collects at most n = Õ( 2 ¯ ¯ A )2 ) samples for each (1−∆A ) ϵPR (1−∆ (s, a) ∈ (A, AA ). Under the event E and Assumption 7.1 then with probability 1 − δ, the η-greedy policy π w.r.t. v ′ (Alg. 7, Line 5) is gain optimal and probability optimal: |gπ − gπA | < ϵPR , P [π |= φ|s0 ∈ A] = 1. Proof of Prop 7.3.2 & Prop E.2.5. Let πv′ be the greedy policy with respect to v ′ . Let gπ̃P̃A be the gain of the gain-optimal policy, π̃A , in A with respect to dynamics P̃ . For the approximation error, 0 ≤ gπP − gπPA = gπP − gπP̃ + gπP̃ − gπP̃v′ + gπP̃v′ − g∗P̃ + g∗P̃ − gπPA ≤ |gπP − gπP̃ | + |gπP̃ − gπP̃v′ | + |gπP̃v′ − gπ̃P̃A | + gπ̃P̃A − gπPA | < {z (a) } | {z (b) } | {z (c) } | {z (d) } ϵPR ϵPR ϵPR + + + 0 = ϵPR . 3 3 3 We have the first inequality because πA is gain optimal in P . By the Simulation Lemma E.2.8 we have that (a) < ϵ3PR by setting ϵ(2) = ϵ3PR in the Lemma. By the η-greedy approximation Lemma E.2.7 we have (b) < ϵ3PR by setting ϵ(1) = ϵ3PR in the Lemma. For (c), since πv′ represents the approximately optimal policy in P̃ L then, by value iteration approximation guarantees, (c) = |gπP̃v′ − gπ̃P̃A | < ϵ2PR ≤ ϵ3PR by setting ϵLPR = 2ϵ3PR [66]. It is known that, by optimism and the aperiodicity transformation [66, 94] for the average cost Bellman operator, gπ̃P̃A < gπPA implying (d) < 0. For the probability of satisfaction, when s0 ∈ A, following a policy that samples every action in AA with positive probability makes the Markov Chain Pπ 217 recurrent. Thus, each s ∈ A is visited infinitely often. In particular there is some s∗ ∈ A visited infinitely often, implying π |= φ. Convergence is guaranteed by Lemma E.2.8: since ρ is halved every iteration ¯ A) ¯ A) ϵ (1−∆ (1−∆ then ρ never falls below (2) , which is reached after log 1 ( ϵPR6|A|c ) = 2|A|cmax max 2 6|A|cmax ϵPR log2 ( ϵPR (1−∆¯ ) ) iterations (since ϵ(2) = 3 ). Further by Lemma E.2.8, we get A 2 2 cmax the sample complexity n = Õ( ϵ2|A| ¯ )2 ), completing the proof. (1−∆ PR A Corollary E.2.6. Under the same assumptions as Prop E.2.5, in addition, |gπP − gbπ̃P̃A | ≤ 4ϵ3PR . Proof. Continuing the same argument as in Prop E.2.5, we have ϵL 4ϵPR 0 ≤ gπP − gπ̃P̃A + gπ̃P̃A − gbπ̃P̃A ≤ ϵPR + PR = , 2 3 where we use triangle inequality and appeal to Prop E.2.5 for |gπP − gπ̃P̃A | ≤ ϵPR L and [66] where |gπ̃P̃A − gbπP̃A | ≤ ϵ2PR ≤ ϵ3PR since ϵLPR = 2ϵ3PR . Lemma E.2.7. (η-greedy approximation) Let P be any dynamics. Let π be a greedy policy in AMEC (A, AA ) with dynamics P . With 0 ≤ η ≤ 1, let πη be η-greedy with respect to π. Then, for any error ϵ(1) > 0, there exists some threshold η ∗ ∈ (0, 1] such that when η ∈ (0, η ∗ ] we have |gπP − gπPη | ≤ ϵ(1) . (E.5) Proof. Let s0 , . . . , s|A|−1 be any ordering of the states in A. The standard (nonoptimistic) average cost Bellman equation with known dynamics P is given   by Lv(s) = g(s) + mina∈AA (s) C(s, a) + P (s, a)v for each s ∈ A for a unique g and v unique up to a constant translation [21]. Furthermore, since the end components are communicating sets then we know that g is a constant vector, i.e. g = g(s) = g(s′ ) for any s, s′ ∈ A [21]. Since v is unique up to translation, we can always set v(0) = 0 to make v unique. The evaluation equations, under h policy π, is similarly, Lπ v(s) = gπ + Ea∼π C(s, a)] + Pπ (s, a)v [21]. For more generality, instead of Pπ we consider αPπ + (1 − α)I, an aperiodicity transform 218 with any coefficient α ∈ [0, 1]. Then the Lπ written as a system takes the form:   v  0   .   ..   αPπ − (1 − α)I 02|A| =  C | {z Xπ    Ea∼π C(s0 , a)    ..   .            −I  v|A|−1  Ea∼π C(s|A|−1 , a)    −     0 D  g0       } ..   ..     .   .     0 g|A|−1 | {z y } | {z bπ } with 0 ...  ...   1 −1 ...  ,D =   ... ...    1  0 0 .. . . . .   C = 0    0 1 0  0   .    −1 This system combines Lv(s) = Ea∼π(s) [C(s, a)] + (αPπ + (1 − α)I)v together with g(s) = g(s′ ) for any s, s′ ∈ A and v(0) = 0. Succinctly, Xπ y − bπ = 0. Similarly, we have Xπη y ′ − bπη = 0. Let dX = Xπη − Xπ , db = bπη − bπ and dy = y ′ − y then (Xπ + dX)(y + dy) − (bπ + db) = 0. Hence, dy = (Xπ + dX)−1 (db − dXy) = (I + Xπ−1 dX)−1 Xπ−1 (db − dXy). We calculate ∥dX∥∞ : ∥dX∥∞ = max |αPπη (s, s′ ) − αPπ (s, s′ )| X s∈A s′ ∈A = max X s∈A |α((1 − η)Pπ (s, s′ ) + ηPU nif (s, s′ )) − αPπ (s, s′ )| s′ ∈A = αη max s∈A X |PU nif (s, s′ ) − Pπ (s, s′ )| s′ ∈A ≤ αη2|A|. By a similar argument, together with C ≤ cmax , then ∥db∥∞ ≤ 2ηcmax Hence, ∥dy∥∞ ≤ ∥(I + Xπ−1 dX)−1 ∥∞ ∥Xπ−1 ∥∞ (∥db∥∞ + ∥dX∥∞ ∥y∥∞ ) ∥Xπ−1 ∥∞ (∥db∥∞ + ∥dX∥∞ ∥y∥∞ ) 1 − ∥Xπ−1 ∥∞ ∥dX∥∞ η∥Xπ−1 ∥∞ ≤ (2cmax + 2α|A|∥y∥∞ ). 1 − 2α|A|η∥Xπ−1 ∥∞ ≤ 219 By selecting ϵ(1) , −1 ∥Xπ ∥∞ (2cmax + 2α|A|∥y∥∞ ) + ϵ(1) 2α|A|∥Xπ−1 ∥∞ we get that ∥dy∥∞ ≤ ϵ(1) and therefore |gπP − gπPη | ≤ ϵ(1) , as desired. η ≤ η∗ = 220 Lemma E.2.8. (Simulation Lemma, Avg. Cost) Fix some α ∈ (0, 1) arbitrary. Let P̃ be the optimistic dynamics achieving the inner minimum of the Bellman equation with respect to LαPR (see Table 7.1) in the AMEC given by (A, AA ). Let π be the η ∗ stochastic policy as in Lemma E.2.7. For some error ϵ(2) > 0. m Let m ∈ N be the smallest value such that ∆((αP̃π + (1 − α)I) ! ) < 1. When n is large enough that ψ(n) ≤ α12  m )) ϵ(2) (1−∆(P̃α,π +1 |A|cmax 1/m −1 then |gπP − gπP̃ | < ϵ(2) . (E.6) ¯ A = maxπ∈Π ∆((αPπ∗ + For ΠA , the set of deterministic policies in A, let ∆ A η m m ∗ (1 − α)I) ) where m = maxπ∈ΠA minm∈N {m|∆((αPπη + (1 − α)I) ) < 1. Then, 2 2 in particular, (E.6) holds after n = Õ( ϵ2 |A|(1−cmax ¯ )2 ) samples are collected for each ∆ state-action pair in (A, AA ). (2) A Proof. Consider, notationally, Pα (s, a, s′ ) = αP (s, a, s′ ) + (1 − α)1{s=s′ } be an aperiodicity transform with α ∈ (0, 1). When fixed by a policy, then Pα,π = αPπ + (1 − α)I. By [166] (Prop. 8.5.8), aperiodicity transforms do not affect gain. Hence gπP = gπPα and gπP̃ = gπP̃α . Let xπ,Pα be the stationary distribution of π in Pα and xπ,P̃α be the stationary distribution of π in P̃α . These quantities exist due to the fact that π has full support over AA making both Pα , P̃α ergodic (finite, irreducible, recurrent, and aperiodic). Hence, |gπP − gπP̃ | = |gπPα − gπP̃α | = |Es∼xπ,Pα [Ea∼π(s) [C(s, a)]] − Es∼xπ,P̃α [Ea∼π(s) [C(s, a)]]| =| X Ea∼π(s) [C(s, a)](xπ,Pα (s) − xπ,P̃α (s))| s∈A ≤ cmax ∥xπ,Pα − xπ,P̃α ∥1 . To bound ∥xπ,Pα − xπ,P̃α ∥1 , we appeal to classic stationary-distribution perturbation bounds [41]. First, since P̃α,π is ergodic then ∃m0 < ∞ such that for any m −P m ∥ ∥P̃α,π α,π ∞ m m ≥ m0 then ∆(P̃α,π ) < 1. Then, in particular, ∥xπ,Pα − xπ,P̃α ∥1 ≤ 1−∆( P̃ m ) α,π [181, 41]. Let E = Pπ,α − P̃π,α , and thus ∥E∥∞ = α∥Pπ − P̃π ∥∞ ≤ α|A|ψ(n). Then, m m m ∥P̃α,π − Pα,π ∥∞ = ∥P̃α,π − (αPπ + (1 − α)I)m ∥∞ m = ∥P̃α,π − (αE + αP̃π + (1 − α)I)m ∥∞ m = ∥P̃α,π − (αE + P̃α,π )m ∥∞ ≤ (α∥E∥∞ + 1)m − 1 ≤ (α2 |A|ψ(n) + 1)m − 1. 221 where in the second-to-last inequality uses that ∥P̃α,π ∥∞ = 1 and ∥AB∥∞ ≤ ∥A∥∞ ∥B∥∞ for matrices A, B. Putting it all together we have that (α2 |A|ψ(n) + 1)m − 1 . (E.7) |gπP − gπP̃ | ≤ cmax m ) 1 − ∆(P̃α,π We therefore require that  m )) 1  ϵ(2) (1 − ∆(P̃α,π ψ(n) ≤ 2 +1 α |A| cmax !1/m  − 1 (E.8) to yield |gπP − gπP̃ | < ϵ(2) . The equation (E.8) also holds with P̃ replaced with P , with (some other) m appropriate. In the AMEC (A, AA ) then there are at most |ΠA | = |A||AA | deterministic policies. For each policy π ∈ ΠA , there is some ηπ∗ satisfying Lemma E.2.7. Let m ¯ A = maxπ∈Π ∆(P m ) < 1 m = maxπ∈ΠA minm∈N {m|∆(Pα,π ) < 1} and ∆ α,πη∗ ∗ A ηπ π (recall this is guaranteed because Pα,πη∗ is ergodic). Then, when ψ(n) < 1 α2 |A|  π ! 1/m ¯ A) ϵ(2) (1−∆ −1 +1 cmax have n = Õ( 2|A| 2 2 m m cmax 2 m (1−∆ ¯ A) m ϵ(2) then |gπP − gπP̃ | < ϵ((2) . By Lemma E.2.2, we 2 2 ) = Õ( ϵ2 |A|(1−cmax ¯ )2 ), since m = 1 achieves the maximum. ∆ (2) A ¯ A < 1. The existence is Remark E.2.9. We do not require knowledge of ∆ sufficient to guarantee convergence. Remark E.2.10. The function ∆(M ), coefficient of ergodicity of matrix M , is a measure (and bound) of the second largest eigenvalue of M . Remark E.2.11. In this chapter, we have assumed that m = 1 and α = 1, for simplicity in exposition. For full rigor, m may be larger, though typically small. m m can be seen as the smallest value making any column of Pα,π dense. From a computational perspective, it is efficient to compute powers of P̃α,π and stop m when P̃α,π has a dense column, making ∆(P̃α,π ) < 1. From there, we can check if ρ (Line 6, Algo 7) satisfies the r.h.s of Eq (E.8). We present the samples required by maximizing over m ∈ N. 222 PlanTransient proofs Proposition 7.3.3 (PlanTransient Convergence & Correctness, Informal). Denote the cost- and prob-optimal policy as π ′ . After collecting at most n = |S\∪k A |2 V̄ 4 Õ( c2i=1ϵ2 iϵ4 ) samples for each (s, a) ∈ (S \ ∪ki=1 Ai ) × A, the greedy policy π min PT φ w.r.t. v ′ (Alg. 8, Line 3) is both cost and probability optimal: ∥Ṽπ − Ṽπ′ ∥ < ϵPT , h i h i |P π reaches ∪ki=1 Ai − P π ′ reaches ∪ki=1 Ai | ≤ ϵφ . Proposition E.2.12 (PlanTransient Convergence & Correctness, Formal). Let {Ai , gi }ki=1 be the set of inputs to Algorithm 8, together with error ϵPT > 0. Denote the cost- and prob-optimal policy as π ′ . After collecting at most n = |S\∪k A |2 V̄ 4 Õ( c2i=1ϵ2 iϵ4 ) samples for each (s, a) ∈ (S \ ∪ki=1 Ai ) × A, under the event E min PT φ and Assumption 7.1 then with probability 1 − δ, , the greedy policy π w.r.t. v ′ (Alg. 8, Line 3) is both cost and probability optimal: ∥Ṽπ − Ṽπ′ ∥ < ϵPT , h i h i |P π reaches ∪ki=1 Ai − P π ′ reaches ∪ki=1 Ai | ≤ ϵφ . Proof of 7.3.3. Convergence follows from boundedness of ∥v∥ ≤ V̄ , and monotone convergence and is well studied [166, 94, 199, 66]. P Fix λ > 0 and drop it from the notation Vπ,λ . Let Ṽ∗P̃ be the value function for the optimal policy in P̃ . For the approximation error, we have 0 ≤ ṼπP − Ṽ∗P = ṼπP − ṼπP̃ + ṼπP̃ − Ṽ∗P < ϵPT . | {z (a) } | {z (b) } For (a) we appeal to Lemma E.2.15 and set ϵ(3) = ϵPT /2 requiring that ψ(n) = |S\∪k Ai |V̄ 2 2 ϵPT cmin , occuring when n = Õ(( ϵPTi=1 ) ) samples per stateϵ2 cmin 14|S\∪k Ai |V̄ 2 (1+ 1 )2 i=1 φ ϵφ action pair have been collected. For (b), by Lemma E.2.16, by selecting ϵLPT = cmin4ϵV̄PT ϵφ we have that 2ϵLPT )v − V∗P cmin 2ϵL v = PT cmin 2V̄ ϵLPT ϵPT ≤ ≤ . ϵφ cmin 2 VπP̃ − V∗P ≤ (1 + For the probability of satisfaction, by Prop E.2.13, we have that π and π ∗ coincide in probability of reaching the states in ∪ki=1 Ai . Proposition E.2.13 (Selecting a bound on ∥v∥). Let {Ai , gi }ki=1 be the set of inputs to Algorithm 8. Let π ′ have maximal probability of reaching 223 ∪ki=1 Ai . Then, with error ϵφ > 0, bounding ∥v∥∞ = ∥LPT v∥∞ ≤ ϵV̄φ where   |S|   1−β 1 + λ cmax guarantees that π returned by Algorithm 8 is near V̄ ≥ β |S| 1−β probability optimal: |P [π |= φ] − P [π ′ |= φ] | < ϵφ . Proof of E.2.13. Suppose V̄ ≥ Jπ + λcmax for any π ∈ Π. Let ϵV̄φ be chosen as upper bound on ∥v∥ = ∥LPT v∥. Denote P [π |= φ] as p, and P [π ′ |= φ] as p∗ . Suppose, for contradiction, p∗ − p > ϵφ , yet π is returned by the Algorithm. This would imply that Ṽπ ≤ Ṽπ′ . Hence, 0 ≤ Ṽπ′ − Ṽπ ≤ p∗ (Jπ′ + λ | k X p∗i i=1 p gbP̃ ) − p(Jπ + λ ∗ π̃A i {z } ≤Jπ +λcmax | k X pi i=1 p V̄ | {z } ϵφ gbπ̃P̃A ) + (p − p∗ ) {z i } ≥0 <−ϵφ < Jπ + λcmax − V̄ ≤ 0. Hence, we have a contradiction. Thus, |p∗ − p| ≤ ϵφ if V̄ ≥ Jπ + λcmax for any π ∈ Π. In fact, since the solution to LPT is deterministic, it suffices to consider only deterministic Π. We will now bound Jπ = Eτ ∼Tπ " Pκπ # t=0 C(st , π(st )) τ |= φ ≤ cmax Eτ ∼Tπ [κτ |τ |= φ], as this is the only unknown quantity. Here Eτ ∼Tπ [κτ |τ |= φ] is the expected number of steps it takes π to leave the transient states. This means that a worst-case bound would be a policy that remains in the transient states as long as possible. We construct the worst-case scenario and give a justification, a formal proof follows from induction. Suppose the starting state is s0 . If π induces a prob-1 transition back to s0 then s0 is recurrent, and so κτ would be small. Instead, π induces a prob 1 − β transition to s0 and a prob β transition to s1 . Notice that the transition to s1 must be at least probability β due to Assumption 7.1. Again, if s1 gave all of its probability to s1 or s0 then a MEC would form and strictly decrease κτ . This process repeats until we reach state s|S|−1 , which has to have a self-loop. If it does not, then, again a large MEC would form and decrease κτ . Of course, this is the well known chain graph, with easily 1 1−β |S| computable expected hitting time: Eτ ∼Tπ [κτ ] ≤ β |S| . By making s|S|−1 1−β |S| 1 1−β the accepting state, then Eτ ∼Tπ [κτ |τ |= φ] = Eτ ∼Tπ [κτ ] = β |S| achieves 1−β 224 the bound. Any other choice of accepting states would strictly decrease κτ . Hence, we can select ! ! 1 1 − β |S| + λ cmax ≥ Jπ + λcmax , V̄ ≥ β |S| 1−β completing the proof. Remark E.2.14. It may also be possible to empirically estimate Jπ rather than take the bound from Prop E.2.13, considering that we have the structure of P through Pb . We give the high level idea. We know all of the AMECs and rejecting EC, so we have all the transient states (denoted T ). Then for some policy π and P ′ ∈ P, submatrix Qπ (s, s′ ) = Pπ′ (s, s′ ) for s, s′ ∈ T represents the transitions in the transient states. It is well known that Eτ ∼Tπ [κτ ] = ∥(I − Q)−1 ∥∞ . Taking the max over all π ∈ Π, P ′ ∈ P, and finally multiplying by cmax gives a bound on Jπ . Lemma E.2.15. (Simulation Lemma, Transient Cost [198]) Consider an MDP (S, A, . . .). For any two transition functions P ′ , P ′′ ∈ P, policy π, and error ϵ(3) > 0 then 1 ′′ ′ ′ ′′ ∥ṼπP ∥∞ = ∥ṼπP ∥∞ ≤ (1+ )V̄ , ∥ṼπP − ṼπP ∥∞ ≤ ϵφ 2 7|S|V̄ 2 (1 + ϵ1φ )2 ψ(n) cmin ≤ ϵ(3) , 4 occurring after n = Õ( ϵ2|S|ϵ4 V̄c2 ) samples from each state-action pair in S × A. (3) φ min Proof. Direct consequence of the definition of V̄ from Prop E.2.13, application of Lemma 2 from [198] and Lemma E.2.2. Lemma E.2.16. (EVI Bound, [198]) Suppose v is returned by VI with accuracy ϵLPT with Bellman equation LPT (See Table 7.1). Suppose π is greedy with respect to v. If ϵLPT ≤ cmin then, element-wise, 2 2ϵLPT P P̃ v ≤ Ṽπ∗ , v ≤ Ṽπ ≤ (1 + )v. cmin 225 E.3 Conjecture on Sample Complexity As we have proven in Theorem 7.4.1, the optimal policy creates a set of AMECs which coincide with (Ai , Ai )ki=1 . For any potential AMEC, we need to guarantee probabilistic closure. For each state-action pair (s, a) ∈ Ai × AAi we have to sample enough times to guarantee that we have “collected” all of the possible unique transitions (s, a, s′ ). Indeed, this is similar to the famous coupon collection problem, where we want to know how much time it will take to collect all unique transitions (s, a, s′ ). Suppose there are m unique tuples each with probability β = m1 . We can use a Chebyshev-based lower bound: 1 P N > m log m − log( )m ≥ δ. δ h i m Simplifying, we get that P N > m log( δ ) ≥ δ. Thus, the number of transitions needed is 1 1 1 m N = Ω(m log( )) = Ω( log( )) = Ω̃( ). δ β βδ β  Further, [222] show that indeed N ≥ logβ β .  226 E.4 Additional Algorithms In this section we discuss the additional subroutines used in this chapter. We discuss the case where selecting ∪ki=1 Ai as the terminal states for SSP in Algo 6 can fail and an alternative solution. Value Iteration Our version of Value Iteration VI (Algo 17) is a two-in-one version, due to the similarity of Relative VI (used in PlanRecurrent) and SSP (used in PlanTransient). The general idea is that you apply the Bellman Operator L onto your current iterate vn repeatedly until d(vn+1 , vn ) exceeds ϵ. When we wish to find the gain, then VT (terminal states) is empty, and we use shifting by the first value of vn (0) for stability [21]. In other words, we subtract vn (0) from every value of vn . On the other hand, if a set of terminal costs is provided then these represent the set of states that we want to reach through SSP and the value vn (s) = VT (s) is known and must be kept fixed throughout applications of L. The only difference in our application of L over standard Bellman operators is that L is optimistic and has an interior minimization over minp∈P(s,a) pT vn (See Table 7.1). To solve this, minimization we use a modified version from [94] given in Algo 18. The idea of Algo 18 is simple: put all the mass of P̃ onto the lowest possible values of vn while still being consistent with Pb . This is efficient as it requires an ordering over v and then a single pass over the states S̃. The calculated probability p(s̃l ) (see Algo 18) are what we call the optimistic dynamics P̃ (s, a, s̃l ). Algorithm 17 Value Iteration (VI) Require: Optimistic Bellman Operator L, Error Measure d, accuracy ϵ > 0, VT terminal values (optional) 1: Set n = 0, v0 = 0S , v1 = Lv0 2: repeat + 3: n← −1 4: if VT is empty then 5: Shift vn ← vn − vn (0)1 {Relative Value Iteration} 6: else 7: vn (s) ← VT (s) for s ∈ VT {SSP} 8: Apply operator vn+1 , P̃ ← Lvn {Bellman Backup} 9: until d(vn+1 , vn ) > ϵ 10: return vn+1 , vn , P̃ 227 Algorithm 18 InnerMin (for PT/PR) Require: A set of states S̃, current estimate from VI vn , estimates Pb (s, a, ·) for a specific (s, a) pair with s ∈ S̃, errors ψ(n), lower bound β (See Assumption 7.1) 1: Sort S̃ = {s̃1 , s̃2 , . . . , s̃m } according to vn (s̃1 ) ≤ vn (s̃2 ) ≤ . . . ≤ vn (s̃m ), where vn is the current 2: Set  b b   min(1 − β, P (s, a, s̃1 ) + ψ(n)), P (s, a, s̃1 ) ̸∈ {0, 1} p(s̃1 ) = 1, Pb (s, a, s̃1 ) = 1   0, Pb (s, a, s̃1 ) = 0 3: For remaining j > 1, set p(s̃j ) = Pb (s, a, s̃j ) 4: Set l ← m P 5: while s̃j ∈S̃ p(s̃j ) > 1 do 6: Reset  P   max(β, 1 − s̃j ̸=s̃l p(s̃j )), p(s̃l ) = 1,   0, Pb (s, a, s̃l ) ̸∈ {0, 1} Pb (s, a, s̃l ) = 1 Pb (s, a, s̃l ) = 0 7: Decrement l ← l − 1 8: Set P̃ (s, a, s̃) = p(s̃) for each s̃ ∈ S̃ 9: return P̃ (s, a, s̃) Modified Algorithm handling Blocking Failure in Algorithm 6 π1 π2 A1 A2 Figure E.1: Blocking Issue. If A1 is included in the terminal AMECs (the states we want to reach) then once it is reached πA1 is instantiated and A1 becomes recurrent, implying only π1 is considered. However, even though it may be the case that Jπ1 < Jπ2 , we may still have Vπ1 > Vπ2 . This example demonstrates the necessity to pick the terminal AMECs properly, rather than just the union of all AMECs found, to avoid blocking. One of the failure modes of Algorithm 6 is in its selection of which AMECs are the necessary AMECs to reach. In fact, by selecting unnecessary AMECs, the SSP procedure fails to treat some AMECs as transient states when in fact, maybe, lower cost could have been achieved if they were. One way to see this is to consider a single directional chain of AMECS (See Figure E.1). In the figure, two policies can be considered: (1) π1 that reaches for A1 and then starts πA1 when A1 is reached, and (2) π2 that reaches for A2 and then starts πA2 when A2 228 is reached. It may be the case that Vπ2 < Vπ1 despite Jπ2 > Jπ1 , since it requires a longer cost path to reach the desired AMEC. Despite this observation, when A1 is selected as terminal states in the subroutine PlanTransient (Algo 8), we disallow consideration of π2 at all. As explained in the proof of Theorem 7.4.1, whatever AMECs are induced by π ∗ coincide with AM EC = {Ai , AAi }ki=1 . Let Ω = 2AM EC \ ∅, all non-empty subsets of AMECs (possible targets). Since all accepting trajectories of π ∗ land in an AMEC, then another way of looking at π ∗ is: π ∗ = min min Vπ , ω∈Ω π∈Π̃(ω) where Π̃(ω) = {π ∈ Πmax |π(s, a) = πAi (s, a) for s ∈ Ai ∈ ω, a ∈ AAi (s)}, which is a policy class where the only degrees of freedom are outside of ω. In other words, π ∈ Π̃(ω) is followed until the trajectory hits Ai ∈ ω and then πAi is followed thereafter. We will reconcile this failure mode of PlanTransient through a modified, nonblocking, subroutine NoBlockPlanTransient (Algo 19). Algorithm 19 NoBlockPlanTransient (NB-PT) Require: States & gains: {(Ai , gi )}ki=1 , err. ϵPT > 0 1: Set v(s) = ∞ for each s ∈ S. 2: Sample φPT times ∀(s, a) ∈ S × A k 3: for ω ∈ 2{Ai }i=1 \ ∅ do 4: Set VT (s) = λgi for s ∈ Ai ⊆ ω 5: vω′ , vω , P̃ ← VI(LPT , dPT , ϵL PT , VT ) 6: if Es∼d0 [vω′ (s)] < Es∼d0 [v(s)] then 7: Set v = vω′ 8: Set π ←greedy policy w.r.t v 9: return π The proof of correctness follows from the fact that vω′ closely tracks Vπ where π is greedy wrt vω′ . Then, selecting the smallest Vπ coincides with Vπ∗ . Proposition E.4.1 (Correctness and Convergence of NoBlockPlanTransient). 2 4 After collecting at most n = Õ( c2|S| ϵV̄2 ϵ4 ) samples for each (s, a) ∈ S × A, under min PT φ the event E and Assumption 7.1 then with probability 1 − δ, , the greedy policy π w.r.t. v ′ (Alg. 8, Line 3) is both cost and probability optimal: 3ϵPT ∥Ṽπ − Ṽπ∗ ∥ < , |P [π |= φ] − P [π ∗ |= φ] | ≤ ϵφ . 2 Proof. Suppose vω < vω′ for any ω ′ ∈ Ω, with ω ∈ Ω. Fix some ω ′ . Denote the greedy policies πvω , πvω′ wrt vω , vω′ . Suppose Ṽπvω′ < Ṽπvω . Then an error was 229 made and 0 ≤ ṼπPvω − ṼπPv ′ ω ≤ ṼπPvω − ṼπP̃vω + ṼπP̃vω − vω + vω − vω′ + vω′ − ṼπP̃v ′ + ṼπP̃v ′ − ṼπPv ′ ω ω ω ϵPT ϵPT ϵPT + +0+0+ 2 2 2 3ϵPT , ≤ 2 where the second line comes from grouping each pair of elements from the first line and applying the bounds found in proof of Proposition E.2.12. ≤ h i On the other hand, suppose p + ϵφ = P [πvω |= φ] + ϵφ < P πvω′ |= φ = p′ . The same proof as in Prop E.2.13 applies to show that the probability of satisfaction remains close: 0 ≤ Ṽπvω′ − Ṽπvω ≤ p′ (Jπvω′ + λ | k X p′i i=1 p gb ) − p(Jπvω + λ ′ π̃Ai {z ≤Jπ +λcmax } | k X pi i=1 p {z ≥0 V̄ | {z } ϵφ gbπ̃Ai ) + (p − p′ ) } <−ϵφ < Jπ + λcmax − V̄ ≤ 0, showing that |p − p′ | < ϵφ . In particular, since the choice of ω ′ was arbitrary, it holds for ω ′ achieving ω ′ = minω∈Ω minπ∈Π̃(ω) Vπ . Therefore the previous bounds all hold for with p′ replaced with p∗ and ṼπPv ′ replaced with ṼπP∗ . ω It is clear we can think of this non-blocking subroutine as checking the different inputs to Algo 8, which requires φPT (ω) = 14|S\ω|ϵPTV̄c2min , occuring when (1+ 1 )2 ϵφ V̄ 2 2 n = Õ(( ϵ|S\ω| ) ) samples per state-action pair have been collected. Taking 2 PT ϵφ cmin V̄ 2 the maximum over ω ∈ Ω, we have n = Õ(( ϵPT|S| )2 ) samples required for ϵ2φ cmin each state-action pair in S × A. Remark E.4.2. Recall S ∗ is set of accepting states in Product-MDP X . This ∗ subroutine appears to have an exponential runtime in |S ∗ |; Ω is at most 2|S | , which is not related to the typical PAC parameters. In general, Ω is modestly small. Remark E.4.3. While the runtime scales poorly with |S ∗ |, the sample complexity remains PAC. 230 Remark E.4.4. We believe it is possible to bring the runtime of the subroutine to be polynomial in |S ∗ | by leveraging the MEC quotient structure (see [16]), but leave that for future work. 231 E.5 Experiments Environments and Details Figure E.2: Environment Illustrations. (Left) Pacman. φ is for the agent, the red triangle, to eventually collect the food, given by the yellow dot, and always avoid the ghost, the red semicircle with eyes. (Right) Mountain Car (MC). φ is to eventually reach the flag. Pacman. This environment (pictured in Fig E.2 Left) is a 5x8 gridworld. The starting positions of the agent (red triangle), food (yellow circle), and ghost (red semicircle with eyes), are as illustrated in Fig E.2. The agent has 4 cardinal directions at each state in addition to a “do nothing” action. The LTL specification is to eventually reach the food and to forever avoid the ghost “F(food) & G(!ghost)”, where the food state is labelled “food” and the ghost state is labelled “ghost”. Once the food is picked up, it is gone. The ghost chases the agent (following the shortest path) with probability .4 and chooses a random action with probability .6. Though this is an infinite horizon problem, as there is no terminal state, we allow a maximum horizon of H = 100 in our experiments. We track how long the agent has avoided the ghost and whether the agent has picked up the food. To simplify verification, we say the agent has satisfied the spec if the food has been picked up and the ghost has been avoided for all H timesteps. The cost function is defined as 1 everywhere. For the shaped LCRL baseline, we use progression through the LDBA as a “reward”: if the agent progresses to a new state in the automaton then the cost of that transition is .1 instead of 1. The authors of LCRL used similar ideas in their code as well. However, we must note that progression-based cost shaping eliminates any guarantee of LTL satisfaction. An agent is incentivized to find cycles in the LDBA rather than find an accepting state. In the case when no such cycles exist, then this form of cost shaping can work. Mountain Car This domain (pictured in Fig E.2 Right) is a discretization 232 of the Mountain Car domain from OpenAI [29], with state-space given by tuple (position, velocity) and cost of 1. We discretize the position space into 32 equal size bins and the velocity into 32 geometrically-spaced bins, allowing more granularity around low velocity than high velocity, making 322 bins (states) in the MDP. The starting state is the standard MC starting state, but then placed in the appropriate bin. A bin can be converted back to (pos,vel), for purposes of sampling from P , by uniformly selecting from the valid positions/velocities implied by the bin. The agent has 3 actions: accelerate left, do nothing, accelerate right. The specification is to eventually reach the goal state “F(goal)”, the standard task, where any bin with position beyond the flag position is labelled “goal”. For the shaped LCRL baseline, we use a cost function of c = .1 if the change in position is positive and the agent accelerated right, likewise if the change is negative and the agent accelerated left, otherwise c = 1. This cost function should incentivize the agent to seek actions which make the car go faster. Unlike the previous experiment, here cost-shaping has no effect on the guarantee of LTL satisfaction. Safe Delivery This domain (pictured in Fig 7.1 Right) is a 4-state MDP: (0) start state, (1) sniffed packet, (2) stolen packet (3) delivered packet. In each state, the agent has two actions, A and B. The transition function P in the MDP is given by P (0, A, 1) = 1, P (0, B, 2) = .5, P (0, B, 3) = .5, P (1, A, 3) = 1, P (1, B, 3) = 1, P (2, A, 2) = 1, P (2, B, 2) = 1, P (3, A, 3) = 1, P (3, B, 3) = 1. In other words, choosing action A in the initial state immediately leads to a sniffed packet, which subsequently leads to the packet being delivered by any action. Alternatively, choosing action B in the initial state has a 50 − 50 chance of having the packet stolen or immediately delivered, regardless of action. Once, stolen, it remains stolen. Once delivered, the packet remains delivered, regardless of action. The states are labelled as L(0) = L(3) =“safe”. The specification is to always stay in safe states: “G(safe)”. Let all the costs be 1. The Product-MDP can be seen in Figure 7.2 Right. The probability-optimal and cost-optimal policy is then choosing B is state 0 and then arbitrarily afterward. The maximum probability of satisfying the policy is 50% because 50% of the time the packet gets stolen. Though this is an infinite horizon problem, as there is no terminal state, we allow a maximum horizon of H = 100 in our experiments. Thus, the average number of timesteps 233 should be .5 ∗ H = 50. Similarly to Pacman, for the shaped LCRL baseline, we use progression through the LDBA as a “reward”: if the agent progresses to a new state in the automaton then the cost of that transition is .5 instead of 1. Infinite Loop This environment (pictured in Fig 7.1 Left) is a 2x5 gridworld. The agent starts in the bottom right corner. The agent has 4 cardinal directions at each state in addition to a “do nothing” action. We consider two specifications: φ1 : The LTL specification is to perpetually visit the office (in the top right corner) followed by the coffee room (top left corner): “GF(o & XFc)”, where the office is labelled o and the coffee room is labelled c. The Product MDP is illustrated in Figure 7.2 Center. φ2 : We require the agent to “G((c -> XXXXXo) & (o ->XXXXXc)) & Xo”, (E.9) meaning to get to first get to office in 1 step, then repeatedly reach the coffee room in 5 steps followed by the office in 5 steps. Similarly to Pacman, for the shaped LCRL baseline, we use progression through the LDBA as a “reward”: if the agent progresses to a new state in the automaton then the cost of that transition is .5 instead of 1. Hyperparameters We use the following hyperparameters for our experiments. Each set of hyperparameters was run with 20 seeds, with the exception of Safe Delivery which was run with 40 seeds. Table E.4: Hyperparameters. Param(s) V̄ cmin cmax φ ϵ δ LCRL Params Max Traj len. γ Learning rate Infinite Loop φ1 Infinite Loop φ2 Safe Delivery Pacman MC 50 50 10 100 150 1 1 1 1 1 1 1 1 1 1 GF(o & XFc) See φ2 in (E.9) G(!unsafe) F(food0) & G!ghost Fgoal 3 3 3 3 10 .1 .1 .1 .1 .1 Infinite Loop 100 .99 .95 Infinite Loop 2 100 .99 .95 Safe Delivery 100 .99 .95 Pacman 100 .99 .95 MC 200 .95 .9 234 1.0 0.8 LCP (Ours) 80 40 20 2 3 4 5 6 7 Total Number of Samples (× 103 ) 0.50 0.45 0.40 0.35 50 Better Better 60 Safe Delivery Better Better 0.6 0.4 0.2 0.0 100 LCRL (shaped) LCRL Max (Expected) Timesteps Infinite Loop ϕ2 Prob of Satisfaction Additonal Results 45 40 35 35 40 45 50 55 60 Total Number of Samples Figure E.3: Additional Results. (Left Column) Infinite Loop 2. φ is a specific trajectory that needs to be followed: first get to the office in 1 timestep and then the coffee room in 5 and then back to the office in 5, over and over. (Right) Safe Delivery (Right Column). φ is to always be safe. In this section we examine additional results for the experiments we ran. For the Infinite Loop environment under φ2 , we see (Figure E.3 Left Column) that our method is able to follow the trajectory specified by φ2 even in low sample regimes. The learning signal for LCRL is very poor as the episode terminates extremely quickly if the agent does not get to the next location that it needs to be in within the allotted time. The shaped LCRL only does marginally better, but still struggles to satisfy the LTL with any probability. For the Safe Delivery environment, we see (Figure E.3 Left Right) that our method picks out the probability-optimal policy. LCRL is nearly optimal. The sparsity of this problem is significantly less as the feedback for spec satisfaction verification comes after a single timestep. Interestingly, cost shaping in Safe Delivery performs worse than straight LCRL. This isn’t surprising since, as noted, the verification feedback comes after a single timestep and is more important than any cost-shaping. However, cost-shaping muddles the feedback making shaped LCRL perform worse. We speculate that with only a few hundred or thousand more samples, both LCRL and shaped LCRL would reach the optimal policy. Recall that LCRL and shaped LCRL are not the same as 235 Q-learning, as they operate in the product-MDP rather than the underlying MDP. Thus, these observations are still consistent with our Motivation section (Section 7.1), insisting that Q-learning would have trouble in this environment. Policies (1) (1) (2) (3) Figure E.4: Types of policies for different φ. (Left) Infinite Loop φ1 . φ1 is to go perpetually walk between the office and the coffee room (Right) Infinite Loop φ2 . φ2 is to get to the office in 1 time step then perpetually, take 5 timesteps to get to the coffee room and 5 steps back to the office. In this section we examine the policies induced by different specificity in specifications. In particular, we consider the Infinite Loop environment with two different specifications φ1 , φ2 , see Section E.5, E.5 for a description. For φ1 , we only require that the agent “eventually” navigate between the office and coffee room. The agent is incentivized to stay in place (create a cost-1 cycle) for as long as possible and very infrequently take a random action. Of course, eventually taking random actions will loop the agent between the office and coffee room. This behavior is illustrated in Figure E.4 Left, where the agent is always in LDBA state 1 and takes random actions with low probability and does nothing with high probability. It takes exponential time for the agent to make a single loop between the office and coffee room. On the other hand, we may want the agent to move quickly. In this case, we can be more specific and use specification φ2 . The behavior for an agent satisfying φ2 is illustrated in Figure E.4 Right. The agent gets to LDBA state 2 by first reaching the office in a single time step. Then the agent loops between LDBA states 2 and 3 by reaching the coffee room and office, repeatedly, within the allotted time. If the agent does not reach the office or chair within the allotted 236 time, there is a fourth LDBA state (unpictured) which is a sink denoting failure of the spec. In essence, the LDBA has created the options, or hierarchy, of solving the problem, as noted in Section 7.1. It takes 10 timesteps for the agent to make a single loop between the office and coffee room. Notice that the high level description of the task is unchanged, but the details of how the task is accomplished is much more specific in φ2 rather than φ1 . This demonstrates that writing LTL task specifications is flexible, but requires thought about “how” the task should be accomplished. 237 Appendix F CHAPTER 6 APPENDIX F.1 Experiments Environment Details The environment and experiment details are summarized in Table F.1. Table F.1: Environment Details. Environment Experiment S M AM Dynamics LTL Formula Minecraft Pacman Flatworld 1 Flatworld 2 Flatworld 3 Flatworld 4 Carlo Discrete Discrete Discrete Discrete [0, 1]2 [0, 1]2 [−1, 1]2 Deterministic Stochastic Deterministic Deterministic Deterministic Deterministic Deterministic GF (y & XF (b)) & (G¬r) F (food) & (G¬ghost) F Gy GF (y & XF (r)) & (G¬b) F Gy GF (y & XF (r)) & (G¬b) GF (zone1 & XF (zone2)) & (G¬crash) Q-learning Q-learning Q-learning Q-learning PPO PPO PPO Discrete Discrete R2 R2 R2 R2 R5 Experiment Setup Each experiment is run with 10 random seeds. Results from Figure 8.2 are from an average over the seeds. Q-learning experiments. Let k be the greatest number of jump transitions available in some LDBA state k = maxb∈S B |AB (b)|. Let m = maxs∈S M |AM (s)|. B The neural network Qθ (s) takes as input s ∈ S M and outputs R(m+k)×|S | a (m + k)-dim vector for each b ∈ S B . For our purposes, we consider Qθ (s, b) to be the single (m + k)-dim vector cooresponding to the particular current state of the LDBA b. When S M is discrete then we parametrize Qθ (s, b) as a table. Otherwise, Qθ (s, b) is parameterized by 3 linear layers with hidden dimension 128 with intermediary ReLU activations and no final activation. After masking for how many jump transitions exist in b, we can select arg maxi∈[0,...,|AB (b)|] Qθ (s, b)i the highest Q-value with probability 1 − η and uniform with η probability. Here, η is initialized to η0 and decays linearly (or exponentially) at some specified frequency (see Table F.2). At each episode (after a rollout of length T ), we perform K gradient steps with 238 different batches of size given in Table F.3. We use Adam optimizer [107] with a learning rate also specified by the table. When in a continuous state space, we implement DDQN [86] (rather than DQN) with a target network that gets updated at some frequency specified by Table F.3. Table F.2: Space). Hyperparameters for Q-learning experiments (Discrete Action η η Decay Experiment η0 Min η Type Rate Freq Batch size K (batches) LR Target update T γ Minecraft Pacman Flatworld 1 Flatworld 2 .9 .05 .9 .9 15 15 .99 .999 .95 .95 .3 .4 .8 .8 0 0 .15 .15 Exp Linear Exp Exp 100 400 100 100 128 512 128 128 20 200 5 5 .001 .001 100 100 20 50 PPO experiments. Let k be the greatest number of jump transitions available in some LDBA state k = maxb∈S B |AB (b)|. The neural network fθ (s) takes B as input s ∈ S M and outputs R(k+2)×|S | is a (k + 2)-dim vector for each b ∈ S B . For our purposes, we consider fθ (s, b) to be the single (k + 2)-dim vector cooresponding to the particular current state of the LDBA b. fθ (s, b) is parameterized by 3 linear layers with hidden dimension 64 with intermediary ReLU activations. The first dimension corresponds to sampling a Gaussian action a ∼ N (fθ (s, b)[0], diag(σ 2 )) where σ is initialized to σ0 (see Table F.3) and decays exponentially (at a rate given in the table) every 10 episodes. The remaining k + 1 dimensions (after proper masking to account for the size of |AB (b)| and softmax) represent the probability p = [pa , pϵ0 , . . . , pϵk ] of taking either the MDP action a or a some jump transition ϵi . We sample from a Categorical(p) variable to select whether to return a ∼ N (Tanh(fθ (s, b)[0]), diag(σ 2 )) or a = ϵi for some i. The density can be calculated by multiplying pa by the Gaussian density when a is selected, and pϵi otherwise. For the critic, we have a parametrized network fφ (s, b) → R of 3 linear layers with hidden dimension 64 with intermediary Tanh activations and no final activation. At each episode (after a rollout of length T ), we perform 5 gradient steps with different batches of size given in Table F.3. The importance sampling term 239 in PPO is clipped to 1 ± .4. The critic learning rate is .01. We use Adam optimizer [107] for both the actor and critic. Table F.3: Hyperparameters for PPO experiments (Continuous Action Space). Experiment σ0 σ Decay Rate Min σ Batch size LR Actor T Flatworld 3 1.8 .98 Flatworld 4 1.8 .99 Carlo .5 .999 .3 .1 .3 128 128 16 .001 .001 .0001 20 50 500 240 F.2 Constructing feasible trajectories for policy gradient during rollout Algorithm 20 LCER for Policy Gradient (Option 2) Require: Dataset D. Trajectory τ of length T . 1: Set T0 ← {(s0 , b)|b ∈ B} 2: for (st , at , st+1 ) ∈ τ do 3: Form Tt according to Eq (F.1) 4: Set D ← D ∪ TT 5: return D Suppose we wanted to generate feasible trajectories in realtime while the policy is being rolled out. That is, we have a partial trajectory of the form τt = (s0 , b0 , a0 , . . . , st , bt ) generated by running π in P . Let at = a ∈ A be the t-th action taken by π and st+1 = s′ ∈ M be the next observed state observed in the MDP. Let Tt be the current set of feasible (partial) trajectories at timestep t. Elements τk = (s0 , b0 , a0 , . . . , sk , bk ) ∈ Tt denote k-step (partial) trajectory, not necessarily part of the trajectory observed during the course of a rollout of π. Here, k ≥ t. Then, for each τk ∈ Tt , one of 4 cases holds: Case 1. Action a is not a jump transition (ie. a ∈ AM (sk )) and there are no jump transitions available in bk (AB (bk ) = ∅). Then we can form the concatenation: τk+1 = τk ∪ (a, s′ , bk+1 ) where bk+1 = P B (bk , LM (s′ )). We set Tϵ = ∅. Case 2. Action a is a jump transition and is currently feasible in bk (ie. a ∈ AM (bk )). Then we can form the concatenation τk+1 = τk ∪ (a, s′ , bk+1 ) where bk+1 = P B (bk , a). We set Tϵ = ∅. Case 3. Action a is a not a jump transition (ie. a ∈ AM (sk )), but there is at least one feasible jump transition in bk (ie. AB (bk ) ̸= ∅). Then, in addition to forming τk+1 from Case 1, we have all the possible jumps: Tϵ = {τk ∪ (ϵ, sk , bk+1 , a, s′ , bk+2 )|∀ϵ ∈ AB (bk ),bk+1 = P B (bk , ϵ), bk+2 = P B (bk+1 , at )}. Case 4. Action a is a jump transition is infeasible in bk (ie. a ̸∈ AB (bk )). In this case, we just pass this trajectory. Setting τk+1 = τk and Tϵ = ∅. At the end of iterating over each element of τk ∈ Tt and forming τk+1 and Tϵ , 241 we can update our current set of feasible trajectories: ! Tt+1 = ∪τk ∈Tt (Tt \ {τk }) ∪ {τk+1 } ∪ Tϵ . (F.1) To put this process simply, we are swapping out τk for τk+1 and also adding in any jump transitions if they are available. The algorithm can be seen in Algo 20.