New Frameworks for Structured Policy Learning Citation Le, Hoang Minh (2020) New Frameworks for Structured Policy Learning. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/rz4w-k233. https://resolver.caltech.edu/CaltechTHESIS:06092020-121556493 Abstract Sequential decision making applications are playing an increasingly important role in everyday life. Research interest in machine learning approaches to sequential decision making has surged thanks to recent empirical successes of reinforcement learning and imitation learning techniques, partly fueled by recent advances in deep learning-based function approximation. However in many real-world sequential decision making applications, relying purely on black box policy learning is often insufficient, due to practical requirements of data efficiency, interpretability, safety guarantees, etc. These challenges collectively make it difficult for many existing policy learning methods to find success in realistic applications. In this dissertation, we present recent advances in structured policy learning, which are new machine learning frameworks that integrate policy learning with principled notions of domain knowledge, which spans value-based, policy-based, and model-based structures. Our framework takes flexible reduction-style approaches that can integrate structure with reinforcement learning, imitation learning and robust control techniques. In addition to methodological advances, we demonstrate several successful applications of the new policy learning frameworks. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Structured Policy Learning, Policy Learning, Reinforcement Learning, Imitation Learning, Safe Machine Learning Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Computing and Mathematical Sciences Awards: Amori Doctoral Prize in CMS, 2020. MIT Sloan Conference 2017, Best paper runner up. Amazon Graduate Fellow, 2017. Thesis Availability: Public (worldwide access) Research Advisor(s): Yue, Yisong Thesis Committee: Wierman, Adam C. (chair) Anandkumar, Anima Daumé, Hal, III Yue, Yisong Defense Date: 22 October 2019 Funders: Funding Agency Grant Number Intel Corporation UNSPECIFIED Disney UNSPECIFIED Pimco UNSPECIFIED NSF 1564330 JPL PDF IAMS100224 Bloomberg UNSPECIFIED Northrop Grumman Corporation UNSPECIFIED Amazon UNSPECIFIED NSF 1645832 Raytheon UNSPECIFIED Record Number: CaltechTHESIS:06092020-121556493 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:06092020-121556493 DOI: 10.7907/rz4w-k233 Related URLs: URL URL Type Description http://hoangle.info/ Author Personal research website http://proceedings.mlr.press/v97/le19a.html Publisher (ICML) Article adapted for Ch. 3 https://arxiv.org/abs/1911.06854 arXiv Article adapted for Ch. 4 http://proceedings.mlr.press/v48/le16.pdf Publisher (ICML) Article adapted for Ch. 5 https://papers.nips.cc/paper/9705-imitation-projected-programmatic-reinforcement-learning.pdf Publisher (NeuRIPS) Article adapted for Ch. 6 http://proceedings.mlr.press/v80/le18a.html Publisher (ICML) Article adapted for Ch. 7 http://proceedings.mlr.press/v70/le17a.html Publisher (ICML) Article adapted for Ch. 8 ORCID: Author ORCID Le, Hoang Minh 0000-0002-5521-5856 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 13809 Collection: CaltechTHESIS Deposited By: Hoang Le Deposited On: 11 Jun 2020 22:12 Last Modified: 10 Dec 2020 00:04 Thesis Files Preview PDF - Final Version See Usage Policy. 8MB Repository Staff Only: item control page

New Frameworks for Structured Policy Learning Thesis by Hoang Minh Le In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2020 Defended October 22, 2019 ii © 2020 Hoang Minh Le ORCID: 0000-0002-5521-5856 All rights reserved except where otherwise noted iii ACKNOWLEDGEMENTS First and foremost, I am deeply grateful for having the guidance of my Ph.D. advisor, Yisong Yue. This dissertation would not have been possible without his support and invaluable insights. I have learned a great deal from Yisong over the past 5 years. I have many fond memories of my first conference talk, my first paper submission, my first research proposal all with Yisong generously spending late night hours to help me take the first steps towards being a machine learning researcher. Outside of technical matters, Yisong’s kindness, patience, and encouragement have been an inspiration. Yisong, it has been an honor and privilege to be your Ph.D. student. I would like to thank my Ph.D. committee members, Adam Wierman, Yisong Yue, Anima Anandkumar, and Hal Daumé, for taking the time out of their packed schedules to provide valuable comments and suggestions. I managed to get “research getaways” at Microsoft Research NYC and Disney Research during my Ph.D. years as summer research interns. I would like to thank my collaborators and mentors, Peter Carr, Alekh Agarwal, Nan Jiang, Miro Dudík, Hal Daumé and Rob Schapire for teaching me new research perspectives. I had a fantastic exposure to the diverse research environment at Disney Research lab in Pittsburgh. I was continuously star-struck around the floor of MSR office in NYC, talking to researchers whose work I greatly admired. Both internships were highly rewarding and eye-opening experiences. I also want to thank other collaborators (in no particular order): Cameron Voloshin, Abhinav Verma, Swarat Chaudhuri, Luciana Cendon, Victor Dorobantu, Andrew Taylor, Aaron Ames, Andrew Kang, Jianhui Chen and Patrick Lucey. I have learned tremendously thanks to having such amazing people work with. It has also been fun discussing many ideas and problems with my lab members, Stephan Zheng, Jialin Song, Eric Zhan, Yuxin Chen, Yanan Sui, Joe Marino, Guanya Shi, Angie Liu, Rose Yu, Jeremy Bernstein, Ellen Feldman, and Taehwan Kim. The Department of Computing and Mathematical Sciences at Caltech has given me amazing research environment and access to inspiring faculty members. Coming from previous non-academic work experience, it was invaluable to take the courses from Joel Tropp, Venkat Chandrasekaran, Adam Wierman, Yisong Yue, Houman Owhadi, Thomas Vidick as a first year graduate student. I want to acknowledge my other fellow Caltech students, who made Ph.D. life fun iv and memorable. Thanks Utkan, Corina, Dimitar, Nina, Benson, Sahin, Oguzhan, Halime, Alvita, Dawna, Kyu, Jing, Andrea, and Jenish for many fun nights and laughters. I want to give a shout-out to my Vietnamese soccer team and support group at Caltech, Kien & Phuong, Lam & Uyen, Tri & Tran, Phong & Duong, anh Thang & chi Nhan. I am very grateful to have all of you guys as part of my Caltech family. And to my Caltech home, I am extremely lucky to have my partner Zeynep and my puppy Tobias with me through the Ph.D. journey. You guys have kept me sane, healthy, well-fed, entertained and motivated along the way. None of this would have been possible without you. Thank you for being my home! Finally and most lovingly, I thank my immediate family for their endless love and sacrifice. I am greatly indebted to my loving parents, who did everything to give me amazing opportunities, despite how little they had. It is through their constant support and understanding that I have had such a rich life experience. I thank my sister and my little niece Trang for always being there. v ABSTRACT Sequential decision making applications are playing an increasingly important role in everyday life. Research interest in machine learning approaches to sequential decision making has surged thanks to recent empirical successes of reinforcement learning and imitation learning techniques, partly fueled by recent advances in deep learning-based function approximation. However in many real-world sequential decision making applications, relying purely on black box policy learning is often insufficient, due to practical requirements of data efficiency, interpretability, safety guarantees, etc. These challenges collectively make it difficult for many existing policy learning methods to find success in realistic applications. In this dissertation, we present recent advances in structured policy learning, which are new machine learning frameworks that integrate policy learning with principled notions of domain knowledge, which spans value-based, policy-based, and modelbased structures. Our framework takes flexible reduction-style approaches that can integrate structure with reinforcement learning, imitation learning and robust control techniques. In addition to methodological advances, we demonstrate several successful applications of the new policy learning frameworks. vi PUBLISHED CONTENT AND CONTRIBUTIONS Jianhui Chen, Hoang M Le, Peter Carr, Yisong Yue, and James J Little. Learning online smooth predictors for realtime camera planning using recurrent decision trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4688–4696, 2016. URL http://openaccess.thecvf.com/content_cvpr_2016/html/Chen_ Learning_Online_Smooth_CVPR_2016_paper.html. H.M.L participated in developing the algorithm, implemented and analyzed the method, conducted the experiments, and participated in the writing of the manuscript. Hoang Le, Andrew Kang, Yisong Yue, and Peter Carr. Smooth imitation learning for online sequence prediction. In International Conference on Machine Learning, pages 680–688, 2016. URL http://proceedings.mlr.press/v48/le16. html. H.M.L participated in the formulation of the project, performed theoretical analysis, implemented and analyzed the method, conducted the experiments, and participated in the writing of the manuscript. Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In International Conference on Machine Learning, pages 2923–2932, 2018. URL http://proceedings. mlr.press/v80/le18a.html. H.M.L participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and participated in the writing of the manuscript. Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703–3712, 2019. URL http://proceedings.mlr.press/v97/le19a.html. H.M.L participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments (with the help from Voloshin), and participated in the writing of the manuscript. Hoang M Le, Peter Carr, Yisong Yue, and Patrick Lucey. Data-driven ghosting using deep imitation learning. URL http://www.sloansportsconference. com/wp-content/uploads/2017/02/1671-2.pdf. H.M.L participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and participated in the writing of the manuscript. Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multiagent imitation learning. In International Conference on Machine Learning, pages 1995–2003, 2017. URL http://proceedings.mlr.press/v70/ vii le17a.html. H.M.L participated in the conception of the project, formulated, implemented and analyzed the method, prepared the data, conducted the experiments, and participated in the writing of the manuscript. A. J. Taylor, V. D. Dorobantu, H. M. Le, Y. Yue, and A. D. Ames. Episodic learning with control lyapunov functions for uncertain robotic systems*. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6878– 6884, 2019. URL https://doi.org/10.1109/IROS40897.2019.8967820. H.M.L participated in analyzing the method and writing of the manuscript. Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitationprojected programmatic reinforcement learning. In Advances in Neural Information Processing Systems, pages 15726–15737, 2019. URL https://papers.nips.cc/paper/9705-imitation-projectedprogrammatic-reinforcement-learning.pdf. H.M.L participated in the conception of the project, developed the algorithm, performed theoretical analysis, and participated in the writing of the manuscript. Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.06854, 2019. URL https://arxiv.org/abs/1911.06854. H.M.L participated in the conception of the project, analyzed the data, and participated in the writing of the manuscript. viii TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Published Content and Contributions . . . . . . . . . . . . . . . . . . . . . . vi Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Structured Policy Learning - Definition and Classification . . . . . . 1 1.2 Learning with Value-Based Structure . . . . . . . . . . . . . . . . . 5 1.3 Learning with Policy-Based Structure . . . . . . . . . . . . . . . . . 6 1.4 Learning with Model-Based Structure . . . . . . . . . . . . . . . . . 7 1.5 Overview Discussion on Existing Approaches to Policy Learning . . 8 Chapter II: Policy Learning under Value-Based Constraints . . . . . . . . . . 13 2.1 Policy Learning under Value-Based Constraints . . . . . . . . . . . . 13 2.2 Problem Formulation in the Batch Setting . . . . . . . . . . . . . . . 15 2.3 Algorithms for Batch Policy Learning under Value-Based Constraints 18 2.4 Theoretical Analysis of Proposed Algorithms . . . . . . . . . . . . . 22 2.5 Empirical Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.6 Other Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Chapter III: Off-Policy Value Estimation for Reinforcement Learning . . . . . 31 3.1 Introduction to Off-Policy Value Estimation . . . . . . . . . . . . . . 31 3.2 Overview of Off-Policy Value Estimators . . . . . . . . . . . . . . . 33 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.5 Discussion and Future Directions . . . . . . . . . . . . . . . . . . . 46 Chapter IV: Regularized Learning with Policy-Based Structure (Smooth Imitation Learning) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.2 Formulating the Problem as Functional Regularization . . . . . . . . 51 4.3 Related Work in Imitation Learning . . . . . . . . . . . . . . . . . . 54 4.4 Algorithm for Smooth Imitation Learning Setting . . . . . . . . . . . 56 4.5 Theoretical Analysis of Reduction-Based Algorithm . . . . . . . . . 58 4.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Chapter V: Reduction to Online Learning with Policy-Based Structure (Programmatic Reinforcement Learning) . . . . . . . . . . . . . . . . . . . . 69 5.1 Introduction to Programmatic Reinforcement Learning . . . . . . . . 69 ix 5.2 Policy Learning Problem within the Structured Policy Class . . . . . 71 5.3 Learning Algorithm via Reduction to Online Learning . . . . . . . . 73 5.4 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.7 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . 85 Chapter VI: Hierarchical Imitation and Reinforcement Learning . . . . . . . 86 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.2 Related Work in Imitation and Reinforcement Learning . . . . . . . 87 6.3 Hierarchical Formalism . . . . . . . . . . . . . . . . . . . . . . . . 88 6.4 Hierarchically Guided Imitation Learning . . . . . . . . . . . . . . . 91 6.5 Hierarchically Guided IL / RL . . . . . . . . . . . . . . . . . . . . . 95 6.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Chapter VII: Policy Learning with Latent Model . . . . . . . . . . . . . . . . 103 7.1 Motivating Applications for Latent Structure . . . . . . . . . . . . . 103 7.2 Policy Learning Problem Formulation . . . . . . . . . . . . . . . . . 105 7.3 Policy Learning Algorithm with Structure Learning and Inference . . 107 7.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 7.5 Related Work in Multi-Agent Learning Context . . . . . . . . . . . . 119 7.6 Limitations and Discussions . . . . . . . . . . . . . . . . . . . . . . 120 Chapter VIII: Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . 121 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Appendix A: Appendix to Chapter 2 . . . . . . . . . . . . . . . . . . . . . . 144 A.1 Equivalence between Regularization and Constraint Satisfaction . . . 144 A.2 Convergence Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . 147 A.3 End-to-end Generalization Analysis of Main Algorithm . . . . . . . 149 A.4 Preliminaries to Analysis of Fitted Q Algorithms . . . . . . . . . . . 153 A.5 Analysis of Fitted Q Evaluation . . . . . . . . . . . . . . . . . . . . 157 A.6 Analysis of Fitted Q Iteration . . . . . . . . . . . . . . . . . . . . . 168 A.7 Additional Instantiation of Meta-Algorithm . . . . . . . . . . . . . . 175 A.8 Additional Experimental Details . . . . . . . . . . . . . . . . . . . . 177 Appendix B: Appendix to Chapter 3 . . . . . . . . . . . . . . . . . . . . . . 181 B.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 B.2 Ranking of Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 183 B.3 Challenging Common Wisdom - Supporting Data . . . . . . . . . . 188 B.4 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 B.5 Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 B.6 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 196 B.7 Additional Supporting Figures for Chapter 3 . . . . . . . . . . . . . 203 B.8 Additional Supporting Tables to Chapter 3 . . . . . . . . . . . . . . 213 Appendix C: Appendix to Chapter 4 . . . . . . . . . . . . . . . . . . . . . . 216 C.1 Detailed Theoretical Analysis and Proofs . . . . . . . . . . . . . . . 216 C.2 Imitation Learning With Smooth Regression Forests . . . . . . . . . 225 Appendix D: Appendix to Chapter 5 . . . . . . . . . . . . . . . . . . . . . . 231 x D.1 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 231 D.2 Additional Experimental Results and Details . . . . . . . . . . . . . 245 Appendix E: Appendix to Chapter 6 . . . . . . . . . . . . . . . . . . . . . . 249 E.1 Proofs for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . 249 E.2 Additional Experimental Details . . . . . . . . . . . . . . . . . . . . 249 E.3 Additional Related Work . . . . . . . . . . . . . . . . . . . . . . . . 256 Appendix F: Appendix to Chapter 7 . . . . . . . . . . . . . . . . . . . . . . 258 F.1 Variational Inference Derivation for Hidden Markov Models . . . . . 258 F.2 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 264 xi LIST OF ILLUSTRATIONS Number Page 1.1 Agent-Environment Interaction Abstraction . . . . . . . . . . . . . . 8 2.1 Empirical Results of Batch Policy Learning under Value-Based Constraints - FrozenLake . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2 Empirical Results of Batch Policy Learning under Value-Based Constraints - CarRacing . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.1 Categorization of Off-Policy Value Estimation Methods . . . . . . . 33 3.2 OPE Method Selection Decision Tree . . . . . . . . . . . . . . . . . 41 3.3 OPE - Compare IPS vs DM vs HM . . . . . . . . . . . . . . . . . . 42 3.4 OPE Various Method Comparison . . . . . . . . . . . . . . . . . . . 44 3.5 OPE Policy Mismatch Comparison . . . . . . . . . . . . . . . . . . 45 4.1 Smooth Policy Learning Illustration . . . . . . . . . . . . . . . . . . 58 4.2 Learning with Smooth Structure vs. Standard Imitation Learning . . 65 4.3 Adaptive vs Fixed Learning Rate - SIMILE . . . . . . . . . . . . . . 65 4.4 Varying Expert Feedback Quality . . . . . . . . . . . . . . . . . . . 66 4.5 Smoothness Structure vs Accuracy Trade-off . . . . . . . . . . . . . 66 4.6 Policy Learning Performance Progression . . . . . . . . . . . . . . . 67 4.7 Deterministic vs. Stochastic Policy Comparison . . . . . . . . . . . . 67 5.1 A High-level Syntax for Programmatic Policies . . . . . . . . . . . . 72 5.2 A Programmatic Policy in TORCS . . . . . . . . . . . . . . . . . . . 72 5.3 Lift-and-Project Policy Learning Diagram . . . . . . . . . . . . . . . 74 5.4 TORCS Experiment - Policy Improvement . . . . . . . . . . . . . . 81 5.5 TORCS Experiment - Number of Crashes . . . . . . . . . . . . . . . 81 6.1 Hierarchical IL RL on Maze Navigation . . . . . . . . . . . . . . . . 97 6.2 Hierarchical IL vs Flat IL on Maze Navigation . . . . . . . . . . . . 97 6.3 Hierarchical IL RL on Montezuma’s Revenge . . . . . . . . . . . . . 100 7.1 Policy Learning with Latent Model - Example . . . . . . . . . . . . 104 7.2 Policy Learning with Latent Model - Algorithm Schematic . . . . . . 107 7.3 Predator-Prey Example . . . . . . . . . . . . . . . . . . . . . . . . . 115 7.4 Policy with and without Structure Comparison . . . . . . . . . . . . 116 7.5 Policy with and without Structure Comparison - Soccer . . . . . . . 117 7.6 Result Visualization - Soccer . . . . . . . . . . . . . . . . . . . . . . 118 xii 7.7 A.1 A.2 B.1 B.2 B.3 B.4 B.5 B.6 B.7 B.8 B.9 B.10 B.11 B.12 B.13 B.14 B.15 B.16 B.17 B.18 B.19 B.20 B.21 B.22 B.23 B.24 B.25 B.26 B.27 B.28 B.29 B.30 B.31 B.32 Latent Role Visualization . . . . . . . . . . . . . . . . . . . . . . . 119 FrozenLake and CarRacing Environment . . . . . . . . . . . . . . . 177 Regularized Policy Learning Baselines . . . . . . . . . . . . . . . . 178 Graph Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Graph-MC Environment . . . . . . . . . . . . . . . . . . . . . . . . 194 Mountain Car Environment . . . . . . . . . . . . . . . . . . . . . . 194 Enduro Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Discrete POMDP Environment . . . . . . . . . . . . . . . . . . . . 194 Gridworld Environment . . . . . . . . . . . . . . . . . . . . . . . . 194 Enduro DM vs IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 MountainCar Comparison - Function Approx . . . . . . . . . . . . . 203 Enduro DM vs HM . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Direct Method Comparison - Small Policy Mismatch, Deterministic Env . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Graph MDP Comparison - Large Policy Mismatch, Stochastic Env . . 204 Compare DM vs DR - Stochastic Env, Large Policy Mismatch . . . . 205 Compare FQE, IH and WIS in Limited Data Regime in Gridworld . . 205 IPS Comparison with IH - Dense and Sparse Rewards . . . . . . . . 205 Exact vs Estimated Behavior Policy Comparison - Pixel Gridworld . 206 Exact vs Estimated Behavior Policy Comparison - Pixel Gridworld . 206 Hybrid Method Comparison . . . . . . . . . . . . . . . . . . . . . . 206 Hybrid Method Comparison . . . . . . . . . . . . . . . . . . . . . . 207 Hybrid Method Comparison . . . . . . . . . . . . . . . . . . . . . . 207 Hybrid Method Comparison . . . . . . . . . . . . . . . . . . . . . . 207 Hybrid Method Comparison . . . . . . . . . . . . . . . . . . . . . . 208 Class Comparison with Unknown Behavior Policy . . . . . . . . . . 208 Class Comparison with Unknown Behavior Policy . . . . . . . . . . 208 Class Comparison with Unknown Behavior Policy . . . . . . . . . . 209 Model-based vs Hybrid Method Comparison . . . . . . . . . . . . . 209 FQE vs Hybrid Method Comparison . . . . . . . . . . . . . . . . . . 209 MRDR vs Hybrid Method Comparison . . . . . . . . . . . . . . . . 209 Q-Regression vs Hybrid Method Comparison . . . . . . . . . . . . . 210 𝑄 𝜋 p𝜆q vs Hybrid Method Comparison . . . . . . . . . . . . . . . . . 210 Retracep𝜆q vs Hybrid Method Comparison . . . . . . . . . . . . . . 210 Tree-Backup vs Hybrid Method Comparison . . . . . . . . . . . . . 210 Doubly Robust Comparison - Pixel Gridworld . . . . . . . . . . . . . 211 xiii B.33 B.34 B.35 B.36 B.37 D.1 D.2 E.1 E.2 E.3 E.4 E.5 E.6 F.1 Weighted Doubly Robust Comparison - Pixel Gridworld . . . . . . . 211 MAGIC Comparison - Pixel Gridworld . . . . . . . . . . . . . . . . 211 Doubly Robust Comparison - Pixel Gridworld . . . . . . . . . . . . . 212 WDR Comparison - Pixel Gridworld . . . . . . . . . . . . . . . . . 212 MAGIC Comparison - Pixel Gridworld . . . . . . . . . . . . . . . . 212 Policy Improvement in MountainCar . . . . . . . . . . . . . . . . . 248 Policy Improvement in Pendulum . . . . . . . . . . . . . . . . . . . 248 Hierarchical IL RL vs Hierarchical RL - Maze Navigation . . . . . . 250 Maze Domain Examples . . . . . . . . . . . . . . . . . . . . . . . . 251 Montezuma’s Revenge Hierarchical Decomposition . . . . . . . . . . 252 Hierarchical IL RL vs Hierarchical RL - Montezuma’s Revenge . . . 253 Learning Progression for Montezuma’s Revenge . . . . . . . . . . . 253 Number of High Level Expert Labels . . . . . . . . . . . . . . . . . 254 Latent Role Assignment Frequency . . . . . . . . . . . . . . . . . . 265 xiv LIST OF TABLES Number Page 3.1 Empirical Environments for OPE . . . . . . . . . . . . . . . . . . . 35 3.2 Empirical Environments for OPE . . . . . . . . . . . . . . . . . . . 35 3.3 OPE Method Selection Guideline - Part 1 . . . . . . . . . . . . . . . 39 3.4 OPE Method Selection Guideline - Part 2 . . . . . . . . . . . . . . . 39 5.1 TORCS Performance Results . . . . . . . . . . . . . . . . . . . . . 82 5.2 Policy Generalization Results in TORCS . . . . . . . . . . . . . . . 83 B.1 Glossary of terms for Chapter 3 . . . . . . . . . . . . . . . . . . . . 182 B.2 Fraction of time among the top estimators across all experiments . . . 183 B.3 OPE - Short Horizon, Small Policy Mismatch . . . . . . . . . . . . . 184 B.4 OPE - Short Horizon, Large Policy Mismatch . . . . . . . . . . . . . 184 B.5 OPE - Long Horizon, Small Policy Mismatch . . . . . . . . . . . . . 185 B.6 OPE - Long Horizon, Large Policy Mismatch, Deterministic Env . . 185 B.7 OPE - Long Horizon, Large Policy Mismatch, Stochastic Env . . . . 186 B.8 OPE - Insufficient Representation . . . . . . . . . . . . . . . . . . . 186 B.9 OPE - Sufficient Representation, Poor Behavior Policy Estimate . . . 187 B.10 OPE - Sufficient Representation, Good Behavior Policy Estimate . . . 187 B.11 Graph MDP, T=10, N=50 . . . . . . . . . . . . . . . . . . . . . . . 188 B.12 Graph MDP, T=100, N=50 . . . . . . . . . . . . . . . . . . . . . . . 188 B.13 Graph MDP, T=10, N=50, Larger Policy Mismatch . . . . . . . . . . 189 B.14 IPS Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 B.15 Graph MDP Env Parameters . . . . . . . . . . . . . . . . . . . . . . 197 B.16 Graph POMDP Env Parameters . . . . . . . . . . . . . . . . . . . . 197 B.17 Gridworld Env Parameters . . . . . . . . . . . . . . . . . . . . . . . 198 B.18 Pixel Gridworld Env Parameters . . . . . . . . . . . . . . . . . . . . 198 B.19 Discrete MountainCar Env Parameters . . . . . . . . . . . . . . . . . 199 B.20 MountainCar Env Parameters . . . . . . . . . . . . . . . . . . . . . 199 B.21 Pixel MountainCar Env Parameters . . . . . . . . . . . . . . . . . . 200 B.22 Enduro Env Parameters . . . . . . . . . . . . . . . . . . . . . . . . 200 B.23 Hyperparameters Choice by Env . . . . . . . . . . . . . . . . . . . . 202 B.24 Graph MDP, 256 trajectories . . . . . . . . . . . . . . . . . . . . . . 213 B.25 Graph MDP, 512 trajectories . . . . . . . . . . . . . . . . . . . . . . 213 B.26 Graph MDP, 1024 trajectories . . . . . . . . . . . . . . . . . . . . . 214 B.27 Graph MDP, 256 trajectories, Dense Rewards . . . . . . . . . . . . . 214 xv B.28 B.29 D.1 D.2 D.3 E.1 E.2 Graph MDP, 512 trajectories, Dense Rewards . . . . . . . . . . . . . 215 Graph MDP, 1024 trajectories, Dense Rewards . . . . . . . . . . . . 215 Performance Results in TORCS . . . . . . . . . . . . . . . . . . . . 246 Generalization Results in TORCS . . . . . . . . . . . . . . . . . . . 247 Performance Results in Classic Control . . . . . . . . . . . . . . . . 247 Network Architecture—Maze Domain . . . . . . . . . . . . . . . . . 252 Network Architecture—Montezuma’s Revenge . . . . . . . . . . . . 255 1 Chapter 1 INTRODUCTION 1.1 Structured Policy Learning - Definition and Classification A Real-World Decision Making Motivation. The goal of this thesis is to make some contributions towards improving learning-based sequential decision making systems that are broadly beneficial for real-world applications. For the purpose of this thesis, we will consider sequential decision making to be the task of making a series of decision in a dynamic environment overtime, to achieve certain specified outcome. While the specific objective of decision making may vary depending on application domain, it is not difficult to appreciate the scope of dynamic decision making systems. Traffic routing, energy and power systems, industrial robots, warehouse logistics, are among long-standing examples of sequential decision making systems that have had large impact on daily lives. Clearly, not all previous examples of decision making systems are what we would currently consider fully algorithmic, or learning-based systems. However, from conversational agents to online recommendation to search and advertising, we are clearly already interacting with increasingly sophisticated learning-based systems on a daily basis. The trends toward more data-driven and algorthmic-driven systems are rooted in exciting developments in machine learning, aided by improving computational power and increasing availability of data. Supervised machine learning has made significant advances in the past decade, with highly visible successes in the areas of computer vision (Krizhevsky et al., 2012; Deng et al., 2009) and natural language processing (Devlin et al., 2019). Powerful deep learning advances have also contributed to recent advances in sequential decision making domains, perhaps most notably computer games (Mnih et al., 2015), and board games (Silver et al., 2017), and robotic manipulations (Andrychowicz et al., 2020). Looking ahead into the near future, researchers and practitioners are exploring potential applications in other areas such as healthcare, remote work and education, autonomous vehicles, automated customer and decision support, etc. Two central questions for these aspirational applications natually arise: (i) how do we approach these diverse problem areas from a machine learning perspective? and (ii) how do we use learning to build real-world systems in a way that is computationally efficient, safe, and grounded in real-world constraints? 2 Policy Learning in Dynamic Environment. This dissertation focuses on the problem of policy learning for sequential decision making tasks. A policy encapsulates the decision making mechanism of an agent interacting with its environment over time. At a high-level, this interaction can be viewed via an abstraction similar to the standard reinforcement learning framework (Sutton and Barto, 2018b; Bertsekas et al., 2005). The protocol is such that at each time step, the environment reveals a context, the agent takes an action given the context, and the action influences the context that the agent observes in the following time step, and so forth. Notice that in general, this abstraction can be more general than a Markov Decision Process (Puterman, 2014). Nevertheless, a policy is simply a mapping from context to (a probability distribution over) actions. Often, the policy learning goal is to derive an optimal policy, where optimality criterion depends on the exact problem formulation. We will give a more in-depth account of contemporary approaches to policy learning in the next chapter. As a rough summary, many recent work in policy learning focus on balancing the exploration-exploitation trade-off, or casting the interaction paradigm under reinforcement / imitation learning dichotomy. When the dynamics are unknown, reinforcement learning (RL) is a general class of samples-based techniques to learn policies, where samples often come from exploring the environment Sutton and Barto (2018b). Efficient exploration is a major challenge of reinforcement learning, and is the subject of very active research (Jiang et al., 2017). Even for multi-arm bandit problem (RL with horizon 1), learning an 𝜖-optimal policy may require 𝑂p 𝜖12 q samples in the worst case (Krishnamurthy et al., 2016). One alternative class of techniques that short-circuit the exploration issue is imitation learning (IL). Imitation learning is a direct way to learn policy by from expert oracle, which can be computational oracle, or human expert. Thus imitation learning can be viewed as generalization of supervised learning to the sequential decision domains. Some Challenges with Contemporary Approaches. Both reinforcement and imitation learning are general purpose classes of methods. However, the success stories of both groups of techniques, while impressive (Cheng et al., 2019c; Pan et al., 2018; Finn et al., 2016; Mnih et al., 2016; Silver et al., 2017), are still constrained to somewhat limited domains, where we either have abundant access to simulated data, or strong assumptions on the oracles. Even with increasing computational power, data efficiency is still paramount. For 3 RL, much recent research effort has been dedicated towards improving sample efficiency of exploration (Kakade et al., 2003; Agarwal et al., 2014; Jiang et al., 2017). Imitation learning generally has been shown to be more sample-efficient than pure exploration-based RL (Sun et al., 2017; Ross and Bagnell, 2014). However, the sample cost to the expert oracle can potentially much more expensive than the sample cost for exploration in scenarios where simulation-to-real transfer is feasible (Peng et al., 2018), or in video games domains where simulation matches reality. The learning agent has to handle imperfect cost and observation. Engineering the right cost (reward) function is difficult for many realistic scenarios. In addition, the most common modeling paradigm in RL is via Markov Decision Processess, which assume full observation (context) of the learning agent. In practice, partial observability is a prevalent setting and policy learning has to account for imperfect information. Finally, we would like to emphasize additional real-world constraints introduced by various domains that are not adequately addressed by classical learning techniques. For policy learning methods to be trusted with realistic applications, they need to meet certain domain-specific desiderata: • One sensible requirement is safety, concerning much of the applications related to human-computer interaction, such as autonomous driving, healthcare, education, etc. (Garcıa and Fernández, 2015) At the most basic level, random exploration is orthogonal to safe learning, thus posing challenges to purely exploration-based RL • Other reasonable constraints may include: stability, which is among the most important concerns for roboticists and control engineers (Berkenkamp et al., 2017); verifiability (or provably correctness in the formal methods sense); smoothness requirement for many human-facing decision making tasks, such as driving; (Bojarski et al., 2016); fairness of policy learning in the context of socio-technical systems (Liu et al., 2018a). Im summary, despite exciting developments, there remains a sizable gap between what current learning-based methods could achieve and broadly useful, real-world applications of policy learning across different domains. Structured Policy Learning. This dissertation describes recent advances that attempt to narrow the gap between contemporary policy learning methods and 4 realistic sequential decision making domains. The proposed methods center around the theme of structured policy learning, which we define as the class of techniques that integrates principled use of domain knowledge into the policy learning process, in order to satisfy real-world desiderata as described above. A natural question given this definition is how should we further clarify the source of domain knowledge? While it is tempting to cast a stereotypical RL technique as assuming “no structure”, and a stereotypical IL technique as assuming a strong form of structure, we argue that in the context of realistic constraints, this distinction is not always the most helpful. Consider two examples: • Consider a hypothetical RL scenario with idealized cost function, where single-stage cost (reward) matches exactly to the consecutive difference (differential) of the optimal reward function. In this setting, optimal cost (reward) design plays the role of the expert Ross and Bagnell (2014) and alleviates the difficulty that is associated with a long horizon problem with sparse cost (reward) signals. While this is an idealized scenario, we can view leveraging off-line (sub-optimal) data, or value specification associated with realistic requirements such as safety, are among possible venues to encode structure into the learning problem. • Consider a hypothetical IL scenario where a computational expert is easy to derive via prior models (e.g., physics-based optimal controller for robotics, or rule-based experts from existing systems). In this setting, perhaps standard imitation learning techniques could succeed in learning the expert’s policy. However, in case of an imperfect model, the cost (reward) corresponding to the expert could be suboptimal in reality. The question facing the learning agent is then how to reconcile and possibly leverage the information coming from the expert in order to move towards another optimality criterion? In examining these two examples above, we can see that integrating the right amount of domain knowledge is not necessarily about forcing a trade-off between RL and IL. In fact, as we will show in this dissertation, in many cases leveraging structure can be a complementary abstraction to improve the learning performance while treating the underlying policy learning sub-routine as a flexible reduction. In this thesis, we will discuss three different ways to integrate structure into policy learning: 5 1. Value-based structure: the desired objective can be expressed via some constraints on the value function of a policy, where the value comes from prior knowledge (e.g., safety), or historical data (e.g., policy improvement). 2. Policy-based structure: the cost (reward) might be difficult to specify, and it can be easier to build the structure more directly into the policy class (e.g., policy is locally stable) 3. Model-based structure: An (approximate) knowledge about the environment or the task is attainable (e.g., known hierarchical structure, or approximate physics-based dynamics model) We aim to incorporate these structures into the design of learning algorithms and show how they can help improve both data efficiency and constraint satisfactions. We will discuss methodological advances as well as empirical validations. For the remainder of this chapter, we briefly introduce several concrete research directions and the corresponding contributions of this dissertation. 1.2 Learning with Value-Based Structure This is a setting where the domain expert can naturally specify constraints on value function, or cost function objectives. In chapter 2, we consider a reinforcement learning scenario where different value-based constraints can be imposed on a policy learning problem in a counterfactual manner, i.e., batch setting. The off-line, off-policy learning approach is promising for realistic applications where existing data is available, but the standard cost objective is not sufficient for learning a reliable decision making agent. We propose a meta learning algorithm, and instantiate with specific subroutines that offer end-to-end finite-sample performance guarantees. The finite-sample results link the number of off-policy samples with how well the algorithm can return an optimal policy subject to satisfying the valuebased constraints. To certify constraint satisfaction, we propose a new off-policy policy evaluation (OPE) algorithm that is applicable in other reinforcement learning settings. Reasoning about value-based structure in a data-efficienct way requires off-policy evaluation of policy performance as a natural component. Off-policy reasoning is highly relevant for any safe and efficient deployment of policy learning to realistic domains. In Chapter 3, we take a detailed look into the empirical performance of different off-policy estimators. We design experimental conditions that highlight 6 different factors that are often neglected in previous work on OPE. Through our analysis, we contribute a summary guideline for method selection. To the best of our knowledge, Chapter 3 is the first comprehensive study of contemporary OPE methods. 1.3 Learning with Policy-Based Structure When the value-based evaluation of the constraints are hard to determined, it can be more natural to build the constraints directly into the policy class. The learning policy can then inherit the structure of the policy class, i.e., the desired structure can be (approximately) “correct-by-construction”, which is an appealing property. For policy-based structure, the challenge lies in designing algorithms that permit reasoning about other optimality criteria of learning. Similar to how regularization encodes prior knowledge in supervised learning context, we take a policy class regularization perspective to policy learning in the sequential setting. In this case, regularizing with respect to a policy class allows mediating between an expressive policy class, such as deep neural networks, and less expressive policy class, but with certifiable properties. In Chapter 4, we regularize to a smooth policy class (under precise notion of smoothness) and designed a learning algorithm that takes a reduction approach to imitation learning. We prove that exploiting this policy structure improve data efficiency of learning. The learning algorithm also outputs learned policy that has better smoothness property than using conventional imitation learning alone. We demonstrate the empirical performance of the algorithm in an realistic applications that require learning smooth decision making policies. In Chapter 5, we generalize this approach to the RL setting where we leverage reduction to online learning, in particular mirror descent, in the policy space. Similar to Chapter 4, regularization to a policy class is enforced via projecting onto a constrained policy space. The projection of the online learning reduction is equivalent to an IL subroutine, where the learning policy learns from the best “expert” from the prescribed policy structure. The update step of online learning can assume the form of standard RL update. In addition to the convergence analysis, in this chapter we demonstrate a potential application in learning “programmatic” policies using RL, where the programs are representable using structured programming language. The algorithm aims to output a near optimal programmatic policy using this liftand-project approach. We show in a driving simulation that the learned policy can 7 generalize better to unseen environment compared to policies learned purely from RL procedures. In addition, programmatic policies are also amenable to formal verification, whereas deep neural networks remain difficult to verify using current techniques. 1.4 Learning with Model-Based Structure Distinct from model-based RL, the model-based structure refers to partial knowledge of the environment or the task, which the learning agent can exploit to help speed up learning process. Below are some examples of model-based structure. Hierarchical structure is well studied in machine learning. In fact, recent success of computer vision can be attributed to good hierarchical feature learning LeCun et al. (1995); Qi et al. (2017). In reinforcement learning context, a common strategy to improve sample efficiency in RL over long time horizons is to exploit hierarchical structure of the problem. In Chapter 6, we propose methods to improve the sample-efficiency of both imitation learning and reinforcement learning, taking into account the hierarchy decomposition. Our key design principle is an algorithmic framework called hierarchical guidance, in which feedback (labels) from the highlevel expert is used to focus (guide) the low-level learner. The high-level expert ensures that low-level learning only occurs when necessary (when subtasks have not been mastered) and only over relevant parts of the state space. This differs from a naïve hierarchical approach which merely gives a subtask decomposition. Focusing on relevant parts of the state space speeds up learning (improves sample efficiency), while omitting feedback on the already mastered subtasks reduces expert effort (improves label efficiency). We validate the approach in long horizon domains, including the challenging Atari game Montezuma Revenge. Our experiments show that incorporating a modest amount of expert feedback can lead to dramatic improvements in performance compared to pure hierarchical RL. Latent structure Reinforcement learning under partial observability is intractable in general POMDP (Papadimitriou and Tsitsiklis, 1987). Recent progress has been made towards policy learning in contextual decision processes under low rank structural assumptions (Jiang et al., 2017). The difficulty of policy learning is compounded for multi-agent settings, where information regarding the intention of other agents in the environment is missing. In Chapter 7, we study the problem of policy learning for multiple coordinating agents from demonstrations. Many realistic multi-agent settings require coordination among collaborative agents to 8 achieve some common goal: modeling team sports, learning policies for game AI, controlling teams of multiple robots, or modeling collective animal behavior. Typically, the agents have access to the outcome of actions from other agents, but the coordination mechanism is neither clearly defined nor observed, which makes the full state only partially observable. We propose a semi-supervised learning framework that integrates and builds upon conventional imitation learning and unsupervised structure learning. The latent structure is represented by a graphical model, which encodes a coordination mechanism of interacting agents. In order to make learning tractable, we develop an alternating optimization method that enables integrated and efficient training of both individual policies and the latent structure model. For learning individual policies, we extend reduction-based single-agent imitation learning approaches into multi-agent domain, utilizing powerful black-box supervised techniques such as deep learning as base routines. For latent structure learning, we develop a stochastic variational inference approach. We show that learning a good latent structure to encode implicit coordination yields significantly superior imitation performance compared to conventional baselines. 1.5 Overview Discussion on Existing Approaches to Policy Learning A sequential decision making agent is characterized by its interaction with the environment (Sutton and Barto, 2018b; Bertsekas et al., 2005). For each time 𝑡, the environment reveals context 𝑥𝑡 in the context space 𝑋, the agent takes an action 𝑎 𝑡 P A, and the action influences the context 𝑥𝑡 `1 that the agent observes in the following time step, and so forth. This interaction paradigm is general, in the sense Figure 1.1: Policy maps context to actions that the context can capture partial observation of the true state, and the agentenvironment interaction can also stand for an interaction between an agent and the expert oracle. As such, this abstraction can convey both the reinforcement learning 9 and imitation learning paradigms. A policy 𝜋 is simply a mapping from context 𝑥 P X to (a probability distribution over) actions in 𝐴. As mentioned in the previous chapter, the optimization objective of policy learning depends on the problem setup. In learning-based approaches, the dynamical system characterizing the agent - environment interaction is typically viewed as unknown or uncertain. If the dynamics are known, one can leverage well-established techniques in dynamic programming and optimal control to solve for optimal policies (Bertsekas et al., 2005). When the dynamics are unknown, reinforcement learning is a general class of sample-based methods to solve optimal control problems (Sutton and Barto, 2018b). Consider a simple and stereotypical example of sequential decision making: a navigation task over a grid where an agent has to make a series of steps towards the goal while avoiding obstacles along the way. Ideally we would want such a learning agent to be able to generalize, i.e., the configuration of the environment and goal locations might change from episode to episode. Without prior knowledge of environmental dynamics, a reinforcement learning agent needs to explore the environment to search for an optimal policy. The objective in RL is usually to minimize long-term cost “ř ‰ (or equivalently, maximizing rewards): 𝐶p𝜋q “ E 𝑡8“0 𝛾 𝑡 𝑐p𝑥𝑡 , 𝑎 𝑡 q , where 𝐶p𝜋q is called the value function of policy 𝜋, and 𝑐p𝑥𝑡 , 𝑎 𝑡 q denotes single-stage cost (reward) associated with observation 𝑥𝑡 and action 𝑎 𝑡 . In the last few years, the combination of deep learning and classical RL techniques have produced very impressive empirical successes mentioned above: from playing Atari games (Mnih et al., 2015), to playing board games (Silver et al., 2017), to certain robotic manipulation tasks (Andrychowicz et al., 2020). Without knowing the dynamics, a learning agent needs to interact with the environment for exploration, before a good policy can be learned via optimization (exploitation). Consequently, success in reinforcement learning is still dependent on having access to environments where simulation data is cheap, as the number of interaction with the environment is significant. Bringing reinforcement learning techniques to real-world applications may introduce several challenges: 1. Well-defined cost objective: The underlying assumption in RL framework is that what the agent optimizes for can be summarized in a well-defined, scalar cost function. Even if such hypothesis is valid, one can imagine designing a good cost function is difficult in many scenarios (e.g., self-driving vehicles). The difficulty of designing cost (reward) function motivates many recent effort to investigate this inverse problem Hadfield-Menell et al. (2017); Finn et al. 10 (2016); Ziebart et al. (2008); Sorg et al. (2010). Designing an optimal cost function can be as hard as solving the forward problem in the worst case Ng et al. (1999). 2. Exploration: Even when we can obtain a reasonable cost function and observe full state, the learning agent may come up against the third issue, which is how to explore the environment efficiently. Efficient exploration is the holygrail of reinforcement learning, and is the subject of very active research (Jiang et al., 2017). Even for multi-arm bandit problem (RL with horizon 1), learning an 𝜖-optimal policy may require 𝑂p 𝜖12 q samples in the worst case (Krishnamurthy et al., 2016). One side effect of random exploration is that the policy may exhibit undesirable properties during training. Given the difficulty of exploration in general, some recent work have developed more efficient exploration algorithms under structural assumptions. (Du et al., 2019) proved an efficient algorithm is possible assuming latent block structure of underlying contextual space. (Misra et al., 2019) built on similar theme and showed that exploiting state abstraction can make hard exploration problems tractable. One alternative class of techniques that short-circuit the exploration issue is imitation learning (IL). Imitation learning is a direct way to learn policy by leveraging domain knowledge. Often the domain knowledge comes from optimal oracle, which can be computational oracle, or human expert. The goal in imitation learning is usually minimize the imitation loss with respect to the expert policy, in that sense the cost function is easy to define: 𝐶p𝜋q “ E𝑑 𝜋 rℓp𝜋p𝑥𝑡 q, 𝜋 ˚ p𝑥𝑡 qqs, where 𝜋 ˚ is the expert policy, and the expectation is taken over state distribution 𝑑 𝜋 induced by the learning policy 𝜋. Many imitation learning techniques involve reductions to supervised learning. Going back to the navigation example, we can obtain a data set from querying oracle policy 𝜋 ˚ , which is either a search or optimization oracle, or human expert to get the shortest path. We can simply reduce the sequential decision problem to supervised learning, e.g., fitting a neural network to directly find the mapping from 𝑋 to 𝐴 in this data set. This simple strategy is also known as behavioral cloning, which was used for one of the first autonomous driving system, ALVINN (Pomerleau, 1989). Behavioral cloning is simple and straightforward to implement, and may in fact work well when data coverage is sufficient. However in many cases, especially under partial observability, a behaviorally cloned policy may fail to generalize to 11 unseen instances of the environment. Interactive imitation learning is an alternative strategy to improve upon this weakness. Specifically, learning proceeds over multiple episodes, similar to many episodic reinforcement learning techniques. At each round of learning, the current policy, which might be performing poorly, can query online expert to yield the correct action labels corresponding to the executed trajectory. With each episode, querying the interactive oracle yields a new data set, which can again be used in some update supervised learning procedure. As such, we reduce the sequential decision problem to multiple supervised learning problems. To update the learning policy, we can use either policy aggregation (Daumé III et al., 2009), or data aggregation, in the style known as DAgger (Ross et al., 2011b). These are among the most common imitation learning techniques that have found widespread successes. Overall, imitation learning can be significantly more sample efficient than pure exploration-based RL Sun et al. (2017). Notice that the two classes of techniques can be complementary: several recent RL success stories rely on bootstrapping the learning agent with a large repository of expert trajectories to increase data and computational efficiency (Silver et al., 2016; Vinyals et al., 2019). Many recent successes from imitation learning are likely unattainable from conventional RL techniques: autonomous driving of a real race car (Pan et al., 2018), visual speech animation (Taylor et al., 2017), or more data-efficient robotic manipulation (Duan et al., 2017). These are examples of tasks that are very difficult to do with deep RL techniques alone, either due to a lack of natural simulation environment, or unacceptable sample efficiency, or both. Still, there are two main drawbacks of imitation learning strategy: 1. Hard to obtain expert labels: Imitation learning can be viewed as generalization of supervised learning to the sequential decision domains. Similar to supervised learning, high-quality labels are required for good learning performance. In many applications, it can be expensive to obtain expert oracle’s labels, especially when interactive expert is part of the learning loop. For computational expert, computational time can be an acceptable trade-off for data acquisition. However, this issue is notable when the agent is learning with human expert in the loop. Imitation learning techniques generally should address the problem of data efficiency for the expert. 2. Non-optimal expert: Second, expert oracle itself may not be optimal, or fully consistent from one expert to another. If the expert policy is not optimal, then 12 the cost objective to learn the policy is also faulty. Here the question of cost function design is similar to the challenge we face in reinforcement learning case. In this section, we give a background overview for conventional approaches to policy learning. In the next chapters, as we develop our approaches to structured policy learning, we will further provide details from relevant recent work that combines policy learning with different forms of structural considerations. 13 Chapter 2 POLICY LEARNING UNDER VALUE-BASED CONSTRAINTS Summary. When learning policies for real-world domains, two important questions arise: (i) how to efficiently use pre-collected off-policy, non-optimal behavior data; and (ii) how to mediate among different competing objectives and constraints. We thus study the problem of batch policy learning under multiple constraints, and offer a systematic solution. We first propose a flexible meta-algorithm that admits any batch reinforcement learning and online learning procedure as subroutines. We then present a specific algorithmic instantiation and provide performance guarantees for the main objective and all constraints. As part of off-policy learning, we propose a simple method for off-policy policy evaluation (OPE) and derive PAC-style bounds. Our algorithm achieves strong empirical results in different domains, including in a challenging problem of simulated car driving subject to multiple constraints such as lane keeping and smooth driving. We also show experimentally that our OPE method outperforms other popular OPE techniques on a standalone basis, especially in a high-dimensional setting. 2.1 Policy Learning under Value-Based Constraints We study the problem of policy learning under multiple constraints. Contemporary approaches to learning sequential decision making policies have largely focused on optimizing some cost objective that is easily reducible to a scalar value function. However, in many real-world domains, choosing the right cost function to optimize is often not a straightforward task. Frequently, the agent designer faces multiple competing objectives. For instance, consider the aspirational task of designing autonomous vehicle controllers: one may care about minimizing the travel time while making sure the driving behavior is safe, consistent, or fuel efficient. Indeed, many such real-world applications require the primary objective function be augmented with an appropriate set of constraints (Altman, 1999). Contemporary policy learning research has largely focused on either online reinforcement learning (RL) with a focus on exploration, or imitation learning (IL) with a focus on learning from expert demonstrations. However, many real-world settings already contain large amounts of pre-collected data generated by existing policies (e.g., existing driving behavior, power grid control policies, etc.). We thus study the 14 complementary question: can we leverage this abundant source of (non-optimal) behavior data in order to learn sequential decision making policies with provable guarantees on both primary objective and constraint satisfaction? We thus propose and study the problem of batch policy learning under multiple constraints. Historically, batch RL is regarded as a subfield of approximate dynamic programming (ADP) (Lange et al., 2012), where a set of transitions sampled from the existing system is given and fixed. From an interaction perspective, one can view many online RL methods (e.g., DDPG (Lillicrap et al., 2016)) as running a growing batch RL subroutine per round of online RL. In that sense, batch policy learning is complementary to any exploration scheme. To the best of our knowledge, the study of constrained policy learning in the batch setting is novel. We present an algorithmic framework for learning sequential decision making policies from off-policy data. We employ multiple learning reductions to online and supervised learning, and present an analysis that relates performance in the reduced procedures to the overall performance with respect to both the primary objective and constraint satisfaction. Constrained optimization is a well studied problem in supervised machine learning and optimization. In fact, our approach is inspired by the work of (Agarwal et al., 2018) in the context of fair classification. In contrast to supervised learning for classification, batch policy learning for sequential decision making introduces multiple additional challenges. First, setting aside the constraints, batch policy learning itself presents a layer of difficulty, and the analysis is significantly more complicated. Second, verifying whether the constraints are satisfied is no longer as straightforward as passing the training data through the learned classifier. In sequential decision making, certifying constraint satisfaction amounts to an off-policy policy evaluation problem, which is a challenging problem and the subject of active research. In this chapter, we develop a systematic approach to address these challenges, provide a careful error analysis, and experimentally validate our proposed algorithms. In summary, our contributions in this chapter are: • We formulate the problem of batch policy learning under multiple constraints, and present the first approach of its kind to solve this problem. The definition of constraints is general and can subsume many objectives. Our approach utilizes multi-level learning reductions, and we show how to instantiate it using various batch RL and online learning subroutines. We show that guarantees from the subroutines provably lift to provide end-to-end guarantees on the 15 original constrained batch policy learning problem. • While leveraging techniques from batch RL as a subroutine, we provide a refined theoretical analysis for general non-linear function approximation that improves upon the previously known sample complexity result (Munos and Szepesvári, 2008). • To evaluate off-policy learning performance and constraint satisfaction, we propose a simple new technique for off-policy policy evaluation (OPE), which is used as a subroutine in our main algorithm. We show that it is competitive to other OPE methods. • We validate our algorithm and analysis with two experimental settings. First, a simple navigation domain where we consider safety constraint. Second, we consider a high-dimensional racing car domain with smooth driving and lane centering constraints. 2.2 Problem Formulation in the Batch Setting We first introduce notation. Let X Ă R𝑑 be a bounded and closed 𝑑-dimensional s be the state space. Let A be a finite action space. Let 𝑐 : X ˆ A ÞÑ r0, 𝐶s s Let there be 𝑚 constraint cost primary objective cost function that is bounded by 𝐶. s each bounded by 𝐺. s To simplify the notation, we functions, 𝑔𝑖 : X ˆ A ÞÑ r0, 𝐺s, s 𝑚 where 𝑔p𝑥, 𝑎q view the set of constraints as a vector function 𝑔 : X ˆ A ÞÑ r0, 𝐺s is the column vector of individual 𝑔𝑖 p𝑥, 𝑎q. Let 𝑝p¨|𝑥, 𝑎q denote the (unknown) transition/dynamics model that maps state/action pairs to a distribution over the next state. Let 𝛾 P p0, 1q denote the discount factor. Let 𝜒 be the initial states distribution. We consider the discounted infinite horizon setting. An MDP is defined using the tuple pX, A, 𝑐, 𝑔, 𝑝, 𝛾, 𝜒q. A policy 𝜋 P Π maps states to actions, i.e., 𝜋p𝑥q P A. The value function 𝐶 𝜋 : X ÞÑ R corresponding to the primary cost function 𝑐 is defined in “ř ‰ the usual way: 𝐶 𝜋 p𝑥q “ E 𝑡8“0 𝛾 𝑡 𝑐p𝑥𝑡 , 𝑎 𝑡 q | 𝑥0 “ 𝑥 , over the randomness of the policy 𝜋 and transition dynamics 𝑝. We similarly define the vector-value function “ř ‰ for the constraint costs 𝐺 𝜋 : X ÞÑ R𝑚 as 𝐺 𝜋 p𝑥q “ E 𝑡8“0 𝛾 𝑡 𝑔p𝑥𝑡 , 𝑎 𝑡 q|𝑥 0 “ 𝑥 . Define 𝐶p𝜋q and 𝐺p𝜋q as the expectation of 𝐶 𝜋 p𝑥q and 𝐺 𝜋 p𝑥q, respectively, over the distribution 𝜒 of initial states. 16 Batch Policy Learning under Constraints In batch policy learning, we have a pre-collected dataset: D “ tp𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐p𝑥𝑖 , 𝑎𝑖 q, 𝑔1:𝑚 p𝑥𝑖 , 𝑎𝑖 qu𝑖𝑛“1 , generated from (a set of) historical behavioral policies denoted jointly by 𝜋D . The goal of batch policy learning under constraints is to learn a policy 𝜋 P Π from D that minimizes the primary objective cost while satisfying 𝑚 different constraints: min 𝐶p𝜋q 𝜋 PΠ s.t. (OPT) 𝐺p𝜋q ď 𝜏 where 𝐺p¨q “ r𝑔1 p¨q, . . . , 𝑔𝑚 p¨qsJ and 𝜏 P R𝑚 is a vector of known constants. We assume that (OPT) is feasible. However, the dataset D might be generated from multiple policies that violate the constraints. Examples of Policy Learning with Constraints Counterfactual & Safe Policy Learning. In conventional online RL, the agent needs to “re-learn” from scratch when the cost function is modified. Our framework enables counterfactual policy learning assuming the ability to compute the new cost objective from the same historical data. A simple example is safe policy learning (Garcıa and Fernández, 2015). Define safety cost 𝑔p𝑥, 𝑎q “ 𝜙p𝑥, 𝑎, 𝑐q as a new function of existing cost 𝑐 and features associated with current state-action pair. The goal here is to counterfactually avoid undesirable behaviors observed from historical data. We experimentally study this safety problem in Section 2.5. Other examples from the literature that belong to this safety perspective include planning under chance constraints (Ono et al., 2015; Blackmore et al., 2011). The constraint here is 𝐺p𝜋q “ ErIp𝑥 P X𝑒𝑟𝑟𝑜𝑟 qs “ Pp𝑥 P X𝑒𝑟𝑟𝑜𝑟 q ď 𝜏. Multi-objective Batch Learning. Traditional policy learning (RL or IL) presupposes that the agent optimizes a single cost function. In reality, we may want to satisfy multiple objectives that are not easily reducible to a scalar objective function. One example is learning fast driving policies under multiple behavioral constraints such as smooth driving and lane keeping consistency (see Section 2.5). Equivalence between Constraint Satisfaction and Regularization Our constrained policy learning framework accommodates several existing regularized policy learning settings. Regularization typically encodes prior knowledge, and has been used extensively in the RL and IL literature to improve learning per- 17 formance. Many instances of regularized policy learning can be naturally cast into (OPT): • Entropy regularized RL (Haarnoja et al., 2017; Ziebart, 2010) maps to policydependent constraint cost 𝑔p𝑥q “ Hp𝜋p¨|𝑥qq, where H measures conditional entropy.1 • Conservative policy improvement (Levine and Abbeel, 2014; Schulman et al., 2015; Achiam et al., 2017) is equivalent to 𝐺p𝜋q “ 𝐷 𝐾 𝐿 p𝜋, 𝜋 𝑘 q, where 𝜋 𝑘 is some “well-behaving” policy. • Smooth imitation learning (Le et al., 2016a) is equivalent to 𝐺p𝜋q “ minℎPH Δpℎ, 𝜋q, where 𝐻 is a class of provably smooth policies and Δ is a divergence metric. • Regularizing RL with expert demonstration (Hester et al., 2018) is equivalent to 𝐺p𝜋q “ Erℓp𝜋p𝑥q, 𝜋 ˚ p𝑥qqs, where 𝜋 ˚ is the expert policy. We provide further equivalence derivation of the above examples in Appendix A.1. Of course, some problems are more naturally described using the regularization perspective, while others using constraint satisfaction. More generally, one can establish the equivalence between regularized and constrained policy learning via a simple appeal to Lagrangian duality as shown in Proposition 2.2.1 below. This Lagrangian duality also has a game-theoretic interpretation (Section 5.4 of (Boyd and Vandenberghe, 2004)), which serves as an inspiration for developing our approach. Proposition 2.2.1. Let Π be a convex set of policies. Let 𝐶 : Π ÞÑ R, 𝐶 : Π ÞÑ R𝐾 be value functions. Consider the two policy optimization tasks: Regularization: min 𝐶p𝜋q ` 𝜆J 𝐺p𝜋q Constraint: min 𝐶p𝜋q s.t. 𝐺p𝜋q ď 𝜏 𝜋 PΠ 𝜋 PΠ Assume that the Slater’s condition is satisfied in the Constraint problem (i.e., D𝜋 s.t. 𝐺p𝜋q ă 𝜏). Assume also that the constraint cannot be removed without changing the optimal solution, i.e., inf 𝜋PΠ 𝐶p𝜋q ă inf 𝜋PΠ:𝐺 p𝜋qď𝜏 𝐶p𝜋q. Then @ 𝜆 ą 0, D 𝜏, and vice versa, such that Regularization and Constraint share the same optimal solutions. (Proof in Appendix A.1.) 1 Constraint value function 𝐺p𝜋q can be viewed as the expectation over discounted state visitation distribution. The lack of explicit discount rate does not intefere with our overall approach. 18 2.3 Algorithms for Batch Policy Learning under Value-Based Constraints To make use of strong duality, we first convexify the policy class Π by allowing stochastic combinations of policies, which effectively expands Π into its convex hull ConvpΠq. Formally, ConvpΠq contains randomized policies, which we denote ř ř 𝜋 “ 𝑇𝑡“1 𝛼𝑡 𝜋𝑡 for 𝜋𝑡 P Π and 𝑇𝑡“1 𝛼𝑡 “ 1. Executing a mixed 𝜋 consists of first sampling one policy 𝜋𝑡 from 𝜋1:𝑇 according to distribution 𝛼1:𝑇 , and then executing ř 𝜋𝑡 . Note that we still have Er𝜋s “ 𝑇𝑡“1 𝛼𝑡 Er𝜋𝑡 s for any first-moment statistic of interest (e.g., state distribution, expected cost). It is easy to see that the augmented version of (OPT) over ConvpΠq has a solution at least as good as the original (OPT). As such, to lighten the notation, we will equate Π with its convex hull for the rest of the chapter. Meta-Algorithm 𝑚 . Clearly The Lagrangian of (OPT) is 𝐿p𝜋, 𝜆q “ 𝐶p𝜋q ` 𝜆J p𝐺p𝜋q ´ 𝜏q for 𝜆 P R` (OPT) is equivalent to the min-max problem: min max 𝐿p𝜋, 𝜆q. We assume (OPT) 𝜋 PΠ 𝜆 PR 𝑘 ` is feasible and that Slater’s condition holds (otherwise, we can simply increase the constraint 𝜏 by a tiny amount). Slater’s condition and policy class convexification ensure that strong duality holds (Boyd and Vandenberghe, 2004), and (OPT) is also equivalent to the max-min problem:max min 𝐿p𝜋, 𝜆q. 𝑘 𝜋 PΠ 𝜆PR` Since 𝐿p𝜋, 𝜆q is linear in both 𝜆 and 𝜋 (due to stochastic mixture2) , strong duality is also a consequence of von Neumann’s celebrated convex-concave minimax theorem for zero-sum games (Von Neumann and Morgenstern, 2007). From a gamethoeretic perspective, the problem becomes finding the equilibrium of a two-player game between the 𝜋´player and the 𝜆´player (Freund and Schapire, 1999). In this repeated game, the 𝜋´player minimizes 𝐿p𝜋, 𝜆q given the current 𝜆, and the 𝜆´player maximizes it given the current (mixture over) 𝜋. We first present a meta-algorithm (Algorithm 1) that uses any no-regret online learning algorithm (for 𝜆) and batch policy optimization (for 𝜋). At each iteration, given 𝜆𝑡 , the 𝜋-player runs Best-response to get the best response: Best-responsep𝜆𝑡 q “ arg min 𝐿p𝜋, 𝜆𝑡 q 𝜋 PΠ “ arg min 𝐶p𝜋q ` 𝜆J 𝑡 p𝐺p𝜋q ´ 𝜏q. 𝜋 PΠ 2 This places no restrictions on the individual policies. Individual policy can be non-linear and cost function can be non-convex. 19 Algorithm 1 Meta-algo for Batch Constrained Learning 1: for each round 𝑡 do 2: 𝜋𝑡 Ð Best-responsep𝜆 q 𝑡ř ř 3: 𝜋p𝑡 Ð 1𝑡 𝑡𝑡 1 “1 𝜋𝑡 1 , 𝜆p𝑡 Ð 1𝑡 𝑡𝑡 1 “1 𝜆𝑡 1 4: Lmax “ max𝜆 𝐿pp 𝜋𝑡 , 𝜆q 5: Lmin “ 𝐿pBest-responsep𝜆p𝑡 q, 𝜆p𝑡 q 6: if Lmax ´ Lmin ď 𝜔 then 7: Return 𝜋p𝑡 8: end if 9: 𝜆𝑡 `1 Ð Online-algorithmp𝜋1 , . . . , 𝜋𝑡 ´1 , 𝜋𝑡 q 10: end for This is equivalent to a standard batch reinforcement learning problem where we learn a policy that is optimal with respect to 𝑐 ` 𝜆J 𝑡 𝑔. The corresponding mixed strategy is the uniform distribution over all previous 𝜋𝑡 . In response to the 𝜋´player, the 𝜆´player employs Online-algorithm, which can be any no-regret algorithm that satisfies: ÿ ÿ 𝐿p𝜋𝑡 , 𝜆𝑡 q ě max 𝐿p𝜋𝑡 , 𝜆q ´ 𝑜p𝑇q 𝜆 𝑡 𝑡 Finally, the algorithm terminates when the estimated primal-dual gap is below a threshold 𝜔 (Lines 7-8). Leaving aside (for the moment) issues of generalization, Algorithm 1 is guaranteed to converge assuming: (i) Best-response gives the best single policy in the class, and (ii) Lmax and Lmin can be evaluated exactly. Proposition 2.3.1. Assuming (i) and (ii) above, Algorithm 1 is guaranteed to stop and the convergence depends on the regret of Online-algorithm. (Proof in Appendix A.2.) Specific Instantiation of Meta-Algorithm We now focus on a specific instantiation of Algorithm 1. Algorithm 2 is our main algorithm in this chapter. Policy Learning. We instantiate Best-response with Fitted Q Iteration (FQI), a model-free off-policy learning approach (Ernst et al., 2005). FQI relies on a series of reductions to supervised learning. The key idea is to approximate the true action-value function 𝑄 ˚ by a sequence t𝑄 𝑘 P F u𝐾𝑘“0 , where F is a chosen function class. 20 Algorithm 2 Constrained Batch Policy Learning Input: Dataset D “ t𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐𝑖 , 𝑔𝑖 u𝑖𝑛“1 „ 𝜋D . Online algorithm parameters: ℓ1 norm bound 𝐵, learning rate 𝜂 𝐵 𝑚 `1 1: Initialize 𝜆 1 “ p 𝑚𝐵 `1 , . . . , 𝑚 `1 q P R 2: for each round 𝑡 do 3: Learn 𝜋𝑡 Ð FQIp𝑐 ` 𝜆J // FQI with cost 𝑐 ` 𝜆J 𝑡 𝑔 𝑡 𝑔q 16: p 𝑡 q Ð FQEp𝜋𝑡 , 𝑐q Evaluate 𝐶p𝜋 p Evaluateř𝐺p𝜋𝑡 q Ð FQEp𝜋𝑡 , 𝑔q 𝜋p𝑡 Ð 1𝑡 𝑡𝑡 1 “1 𝜋𝑡 1 ř ř p 𝜋𝑡 q Ð 1 𝑡 1 𝐶p𝜋 p 𝑡 1 q, 𝐺pp p 𝑡1 q p 𝜋𝑡 q Ð 1 𝑡 1 𝐺p𝜋 𝐶pp 𝑡 “1 𝑡 “1 𝑡 𝑡 ř 𝑡 𝜆p𝑡 Ð 1𝑡 𝑡 1 “1 𝜆 𝑡 1 Learn 𝜋r Ð FQIp𝑐 ` 𝜆pJ 𝑡 𝑔q p p 𝜋 q Ð FQEpr Evaluate 𝐶pr 𝜋 q ЈFQEpr 𝜋 , 𝑐q, 𝐺pr 𝜋 , 𝑔q ˙ ” ıJ p 𝜋𝑡 q ´ 𝜏qJ , 0 p 𝜋𝑡 q ` 𝜆J p𝐺pp pmax “ max L 𝐶pp 𝜆,}𝜆}1 “𝐵 ” ıJ p 𝜋 q ` 𝜆pJ p𝐺pr p 𝜋 q ´ 𝜏qJ , 0 pmin “ 𝐶pr L 𝑡 pmax ´ L pmin ď 𝜔 then if L Return 𝜋p𝑡 end if ” ıJ p 𝑡 q ´ 𝜏qJ , 0 P R𝑚`1 Set 𝑧 𝑡 “ p𝐺p𝜋 17: 𝜆 𝑡`1 r𝑖s “ 𝐵 ř 𝑡𝜆 r 𝑗s𝑒´ 𝜂𝑧𝑡 r 𝑗 s @𝑖 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 𝜆 r𝑖s𝑒´ 𝜂𝑧𝑡 r𝑖 s 𝑗 // Algo 3 with 𝜋𝑡 , cost 𝑐 // Algo 3 with 𝜋𝑡 , cost 𝑔 // FQI with cost 𝑐 ` 𝜆pJ 𝑡 𝑔 // 𝜆r𝑖s the 𝑖 𝑡 ℎ coordinate 𝑡 18: end for In Lines 3 & 9, FQIp𝑐 ` 𝜆J 𝑔q is defined as follows. With 𝑄 0 randomly initialized, r 𝑘 “ tp𝑥𝑖 , 𝑎𝑖 q, 𝑦𝑖 u𝑛 where: for each 𝑘 “ 1, . . . , 𝐾, we form a new training dataset D 𝑖 “1 @𝑖 : 𝑦𝑖 “ p𝑐𝑖 ` 𝜆J 𝑔𝑖 q ` 𝛾 min 𝑄 𝑘 ´1 p𝑥𝑖1 , 𝑎q, 𝑎 and p𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐𝑖 , 𝑔𝑖 q „ D (original dataset). A supervised regression procedure is called to solve for: 𝑛 1ÿ 𝑄 𝑘 “ arg min p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 . 𝑛 𝑓 PF 𝑖 “1 FQI returns the policy: 𝜋 𝐾 “ arg min𝑎 𝑄 𝐾 p¨, 𝑎q. FQI has been shown to work well with several empirical domains: spoken dialogue systems (Pietquin et al., 2011), physical robotic soccer (Riedmiller et al., 2009), and cart-pole swing-up (Riedmiller, 2005), and clinical treatment (Prasad et al., 2017). Another possible model-free subroutine is Least-Squares Policy Iteration (LSPI) (Lagoudakis and Parr, 2003b). One can also consider model-based alternatives (Ormoneit and Sen, 2002). 21 Algorithm 3 Fitted Q Evaluation: FQEp𝜋, 𝑐q 𝑛 Input: Dataset D “ t𝑥 𝑖 , 𝑎 𝑖 , 𝑥𝑖1 , 𝑐 𝑖 u𝑖“1 „ 𝜋D . Function class F . Policy 𝜋 to be evaluated 1: Initialize 𝑄 0 P F randomly 2: for 𝑘 “ 1, 2, . . . , 𝐾 do 3: Compute target 𝑦 𝑖 “ 𝑐 𝑖 ` 𝛾𝑄 𝑘´1 p𝑥 𝑖1 , 𝜋p𝑥 𝑖1 qq @𝑖 r 𝑘 “ tp𝑥 𝑖 , 𝑎 𝑖 q, 𝑦 𝑖 u𝑛 4: Build training set D 𝑖“1 5: Solve a supervised ř learning problem: 𝑛 𝑄 𝑘 “ arg min 𝑛1 𝑖“1 p 𝑓 p𝑥 𝑖 , 𝑎 𝑖 q ´ 𝑦 𝑖 q2 𝑓 PF 6: end for Output: 𝐶p 𝜋 p𝑥q “ 𝑄 𝐾 p𝑥, 𝜋p𝑥qq @𝑥 Off-policy Policy Evaluation. A crucial difference between constrained policy learning and existing work on constrained supervised learning is the technical chalp 𝜆q (Lines lenge of evaluating the objective and constraints. First, estimating 𝐿p𝜋, p p 11-12) requires estimating 𝐶p𝜋q and 𝐺p𝜋q. Second, any gradient-based approach to p Online-learning requires passing in 𝐺p𝜋q ´ 𝜏 as part of gradient estimate (line 15). This problem is known as the off-policy policy evaluation (OPE) problem: we p p need to evaluate 𝐶p𝜋q and 𝐺p𝜋q having only access to data D „ 𝜋D There are three main contemporary approaches to OPE: (i) importance weighting (IS) (Precup et al., 2000, 2001), which is unbiased but often has high-variance; (ii) regression-based direct methods (DM), which are typically model-based (Thomas and Brunskill, 2016b),and can be biased but have much lower variance than IS; and (iii) doubly-robust techniques (Jiang and Li, 2016a; Dudík et al., 2011b), which combine IS and DM. We propose a simple model-free technique using function approximation, called Fitted Q Evaluation (FQE). FQE is based on an iterative reductions scheme similar to FQI, but for the problem of off-policy evaluation. Algorithm 3 lays out the steps. The key difference with FQI is that the 𝑚𝑖𝑛 operator is replaced by 𝑄 𝑘 ´1 p𝑥𝑖1 , 𝜋p𝑥𝑖1 qq (Line 3 of Algorithm 3). Each 𝑥𝑖1 comes from the original D. Since we know 𝜋p𝑥𝑖1 q, r 𝑘 is well-defined. Note that FQE can be plugged-in as a direct method if one each D wishes to augment the policy evaluation with a doubly-robust technique. Online Learning Subroutine. As 𝐿p𝜋𝑡 , 𝜆q is linear in 𝜆, many online convex optimization approaches can be used for Online-algorithm. Perhaps the simpliest choice is Online Gradient Descent (OGD) (Zinkevich, 2003). We include an instantiation using OGD in Appendix A.7. For our main Algorithm 2, similar to (Agarwal et al., 2018), we use Exponenti- 22 ated Gradient (EG) (Kivinen and Warmuth, 1997), which has a regret bound of a ? 𝑂p logp𝑚q𝑇q instead of 𝑂p 𝑚𝑇q as in OGD. One can view EG as a variant of Online Mirror Descent (Nemirovsky and Yudin, 1983) with a softmax link function, or of Follow-the-Regularized-Leader with entropy regularization (Shalev-Shwartz et al., 2012). Gradient-based algorithms generally require bounded 𝜆. We thus force }𝜆}1 ď 𝐵 using hyperparameter 𝐵. Solving (OPT) exactly requires 𝐵 “ 8. We will analyze Algorithm 2 with respect to finite 𝐵. With some abuse of notation, we augment 𝜆 into a p𝑚 ` 1q´dimensional vector by appending 𝐵 ´ }𝜆}1 , and augment the constraint cost vector 𝑔 by appending 0 (Lines 11, 12 & 15 of Algorithm 2).3 2.4 Theoretical Analysis of Proposed Algorithms Convergence Guarantee The convergence rate of Algorithm 2 depends on the radius 𝐵 of the dual variables s and the number of constraints 𝑚. In particular, 𝜆, the maximal constraint value 𝐺, 2 we can show 𝑂p 𝜔𝐵2 q convergence for primal-dual gap 𝜔. Theorem 2.4.1 (Convergence of Algorithm 2). After 𝑇 iterations, the empirical duality gap is bounded by s2 pmax ´ L pmin ď 2 𝐵 logp𝑚 ` 1q ` 2𝜂𝐵𝐺 L 𝜂𝑇 Consequently, to achieve the primal-dual gap of 𝜔, setting 𝜂 “ 4𝐺s𝜔2 𝐵 will ensure that Algorithm 2 converges after s2 logp𝑚 `1q 16𝐵2 𝐺 iterations. (Proof in Appendix A.2.) 𝜔2 Convergence analysis of our main Algorithm 2 is an extension of the proof to Proposition 2.3.1, leveraging the no-regret property of the EG procedure (ShalevShwartz et al., 2012). Generalization Guarantee of FQE and FQI In this section, we provide sample complexity analysis for FQE and FQI as standalone procedures for off-policy evaluation and off-policy learning. We use the notion of pseudo-dimension as capacity measure of non-linear function class F (Friedman et al., 2001). Pseudo-dimension dimF , which naturally extends VC dimension into the regression setting, is defined as the VC dimension of the function class induced by the sub-level set of functions of F : dimF “ VC-dimptp𝑥, 𝑦q ÞÑ 3 The p𝑚 ` 1q𝑡 ℎ coordinate of 𝑔 is thus always satisfied. when executing EG. This augmentation is only necessary 23 signp 𝑓 p𝑥q ´ 𝑦q : 𝑓 P F uq. Pseudo-dimension is finite for a large class of function approximators. For example, (Bartlett et al., 2017) bounded the pseudodimension of piece-wise linear deep neural networks (e.g., with ReLU activations) as 𝑂p𝑊 𝐿 log 𝑊q, where 𝑊 is the number of weights, and 𝐿 is the number of layers. Both FQI and FQE rely on reductions to supervised learning to update the value functions. In both cases, the learned policy and evaluation policy induces a different state-action distribution compared to the data generating distribution 𝜇. We use the notion of concentration coefficient for the worst case, proposed by (Munos, 2003), to measure the degree of distribution shift. The following standard assumption from analysis of related ADP algorithms limits the severity of distribution shift over future time steps: Assumption 1 (Concentrability coefficient of future state-action distribution). (Munos, 2003, 2007; Munos and Szepesvári, 2008; Antos et al., 2008a,b; Lazaric et al., 2010, 2012; Farahmand et al., 2009; Maillard et al., 2010) Let 𝑃 𝜋 be the operator acting on 𝑓 : X ˆ A ÞÑ R s.t. ş p𝑃 𝜋 𝑓 qp𝑥, 𝑎q “ X 𝑓 p𝑥 1 , 𝜋p𝑥 1 qq𝑝p𝑑𝑥 1 |𝑥, 𝑎q. Given data generating distribution 𝜇, initial state distribution 𝜒, for 𝑚 ě 0 and an arbitrary sequence of stationary policies t𝜋𝑚 u𝑚 ě1 define the concentration coeffient: › › › 𝑑p𝜒𝑃 𝜋1 𝑃 𝜋2 . . . 𝑃 𝜋𝑚 q › › › 𝛽 𝜇 p𝑚q “ sup › › 𝑑𝜇 𝜋1 ,...,𝜋 𝑚 8 We assume 𝛽 𝜇 “ p1 ´ 𝛾q2 ř 𝑚𝛾 𝑚 ´1 𝛽 𝜇 p𝑚q ă 8. 𝑚 ě1 This assumption is valid for a fairly large class of MDPs (Munos, 2007). For instance 𝛽 𝜇 is finite for any finite MDP, or any infinite state-space MDP with bounded transition density.4 Having a finite concentration coefficient is equivalent the top-Lyapunov exponent Γ ď 0 (Bougerol and Picard, 1992), which means the underlying stochastic system is stable. We show below a simple sufficient condition for Assumption 1 (albeit stronger than necessary). Example 2.4.1. Consider an MDP such that for any non-stationary distribution 𝜌, 𝜌 p𝑥 q the marginals over states satisfy 𝜇 𝑥𝑥 p𝑥 q ď 𝐿 (i.e., transition dynamics are sufficiently 4 This assumption ensures sufficient data diversity, even when the executing policy is determinis- tic. An example of how learning can fail without this assumption is based on the “combination lock” MDP (Koenig and Simmons, 1996). In this deterministic MDP example, 𝛽 𝜇 can grow arbitrarily large, and we need an exponential number of samples for both FQE and FQI. See Appendix A.4. 24 stochastic), and D𝑀 : @𝑥, 𝑎 : 𝜇p𝑎|𝑥q ą 𝑀1 (i.e., the behavior policy is sufficiently exploratory). Then 𝛽 𝜇 ď 𝐿𝑀. Recall that for a given policy 𝜋, the Bellman (evaluation) operator is defined as ş pT𝜋 𝑄qp𝑥, 𝑎q “ 𝑟p𝑥, 𝑎q ` 𝛾 X 𝑄p𝑥 1 , 𝜋p𝑥 1 qq𝑝p𝑑𝑥 1 |𝑥, 𝑎q. In general T𝜋 𝑓 may not belong to F for 𝑓 P F . For FQE (and FQI), the main operation in the algorithm is to iteratively project T𝜋 𝑄 𝑘 ´1 back to F via 𝑄 𝑘 “ arg min 𝑓 PF } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 }. The performance of both FQE and FQI thus depend on how well the function class F approximates the Bellman operator. We measure the ability of function class F to approximate the Bellman evaluation operator via the worst-case Bellman error: Definition 2.4.1 (inherent Bellman evaluation error). Given a function class F and policy 𝜋, the inherent Bellman evaluation error of F is defined as 𝑑 F𝜋 “ sup𝑔PF inf 𝑓 PF } 𝑓 ´ T𝜋 𝑔}𝜋 where }¨}𝜋 is the ℓ2 norm weighted by the state-action distribution induced by 𝜋. We are now ready to state the generalization bound for FQE: Theorem 2.4.2 (Generalization error of FQE). Under Assumption 1, for 𝜖 ą 0 & 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q Evaluation (Algorithm 3), for 𝑛 “ ˘ ` s4 s2 𝑂 𝐶𝜖 2 plog 𝐾𝛿 ` dimF log 𝐶𝜖 2 ` log dimF q , we have with probability 1 ´ 𝛿: p 𝐶p𝜋q ´ 𝐶p𝜋q ď ˘ 𝛾 1{2 `a ` 𝜋 2𝛾 𝐾 {2𝐶s ˘ ` 𝜖 ` . 𝛽 2𝑑 𝜇 F p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 r 12 q, compared to 𝑂p r 14 q from other related This result shows a dependency on 𝜖 of 𝑂p 𝜖 𝜖 ADP algorithms (Munos and Szepesvári, 2008; Antos et al., 2008b). The price that we pay is a multiplicative constant 2 in front of the inherent error 𝑑 F𝜋 . The error from second term on RHS decays exponentially with iterations 𝐾. The proof is in Appendix A.5. We can show an analogous generalization bound for FQI. While FQI has been widely used, to the best of our knowledge, a complete analysis of FQI for non-linear function approximation has not been previously reported.5 Definition 2.4.2 (inherent Bellman optimality error). (Munos and Szepesvári, 2008) Recall that the Bellman optimality operator is defined as pT𝑄qp𝑥, 𝑎q “ 𝑟p𝑥, 𝑎q ` 5 FQI for continuous action space from (Antos et al., 2008a) is a variant of fitted policy iteration and not the version of FQI under consideration. The appendix of (Lazaric and Restelli, 2011) contains a proof of FQI specifically for linear function class. 25 ş 𝛾 X min𝑎1 PA 𝑄p𝑥 1 , 𝑎 1 q𝑝p𝑑𝑥 1 |𝑥, 𝑎q. Given a function class F , the inherent Bellman error is defined as 𝑑 F “ sup𝑔PF inf 𝑓 PF } 𝑓 ´ T𝑔} 𝜇 , where }¨} 𝜇 is the ℓ2 norm weighted by 𝜇, the state-action distribution induced by 𝜋D . Theorem 2.4.3 (Generalization error of FQI). Under Assumption 1, for 𝜖 ą 0 ` s4 & 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q Iteration, for 𝑛 “ 𝑂 𝐶𝜖 2 plog 𝐾𝛿 ` ˘ s2 dimF log 𝐶𝜖 2 ` log dimF q , we have with probability 1 ´ 𝛿: 𝐶 ˚ ´ 𝐶p𝜋 𝐾 q ď ˘ 2𝛾 `a 𝐾 {2 s p2𝑑 𝛽 ` 𝜖q ` 2𝛾 𝐶 𝜇 F p1 ´ 𝛾q3 where 𝜋 𝐾 is the policy acting greedy with respect to the returned function 𝑄 𝐾 . (Proof in Appendix A.6.) End-to-End Generalization Guarantee We are ultimately interested in the test-time performance and constraint satisfaction of the returned policy from Algorithm 2. We now connect the previous analyses from Theorems 2.4.1, 2.4.2 & 2.4.3 into an end-to-end error analysis. Since Algorithm 2 uses FQI and FQE as subroutines, the inherent Bellman error terms 𝑑 F and 𝑑 F𝜋 will enter our overall performance bound. Estimating the inherent Bellman error caused by function approximation is not possible in general (chapter 11 of (Sutton and Barto, 2018b)). We assume existence of a sufficiently expressive F that can generally make 𝑑 F and 𝑑 F𝜋 arbitrarily small. To simplify our end-to-end analysis, consider 𝑑 F “ 0 and 𝑑 F𝜋 “ 0, i.e., the function class F is closed under applying the Bellman operator. Assumption 2 (Bellman operator realizability). We consider function classes F sufficiently rich so that @ 𝑓 : T 𝑓 P F & T𝜋 𝑓 P F for the policies 𝜋 returned by Algorithm 2. With Assumptions 1 & 2, we have the following error bound: Theorem 2.4.4 (Generalization guarantee of Algorithm 2). Let 𝜋 ˚ be the optimal s Let 𝐾 be the number of iterations of FQE policy to (OPT). Denote 𝑉s “ 𝐶s ` 𝐵𝐺. and FQI. Let 𝜋ˆ be the policy returned by Algorithm 2, with termination threshold 𝜔. ˘ ` s4 s2 𝐾 p𝑚 `1q ` dimF log 𝑉𝜖 2 ` log dimF q , For 𝜖 ą 0 & 𝛿 P p0, 1q, when 𝑛 “ 𝑂 𝑉𝜖 2 plog 𝛿 we have with probability at least 1 ´ 𝛿: 𝐶p𝜋q ˆ ď 𝐶p𝜋 ˚ q ` 𝜔 ` ˘ p4 ` 𝐵q𝛾 `a 𝛽 𝜇 𝜖 ` 2𝛾 𝐾 {2𝑉s , 3 p1 ´ 𝛾q 26 and 𝑉s ` 𝜔 𝛾 1{2 `a 2𝛾 𝐾 {2𝑉s ˘ 𝐺p𝜋q ˆ ď 𝜏`2 ` 𝛽𝜇 𝜖 ` . 𝐵 p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 The proof is in Appendix A.3. This result guarantees that, upon termination of Algorithm 2, the true performance on the main objective can be arbitrarily close to that of the optimal policy. At the same time, each constraint will be approximately satisfied with high probability, assuming sufficiently large 𝐵 & 𝐾, and sufficiently small 𝜖. 2.5 Empirical Study We perform experiments on two different domains: a grid-world domain (from OpenAI’s FrozenLake) under a safety constraint, and a challenging high-dimensional car racing domain (from OpenAI’s CarRacing) under multiple behavior constraints. We seek to answer the following questions in our experiments: (i) whether the empirical convergence behavior of Algorithm 2 is consistent with our theory; and (ii) how the returned policy performs with respect to the main objective and constraint satisfaction. Appendix A.8 includes a more detailed discussion of our experiments. Frozen Lake. Environment & Data Collection. The environment is an 8x8 grid. The agent has 4 actions N,S,E,W at each state. The main goal is to navigate from a starting position to the goal. Each episode terminates when the agent reaches the goal or falls into a hole. The main cost function is defined as 𝑐 “ ´1 if goal is reached, otherwise 𝑐 “ 0 everywhere. We simulate a non-optimal data gathering policy 𝜋D by adding random sub-optimal actions to the shortest path policy from any given state to goal. We run 𝜋 𝐷 for 5000 trajectories to collect the behavior dataset D (with constraint cost measurement specified below). Counterfactual Safety Constraint. We augment the main objective 𝑐 with safety constraint cost defined as 𝑔 “ 1 if the agent steps into a hole, and 𝑔 “ 0 otherwise. We set the constraint threshold 𝜏 “ 0.1, roughly 75% of the accumulated constraint cost of behavior policy 𝜋D . The threshold can be interpreted as a counterfactually acceptable probability that we allow the learned policy to fail. pmax ´ L pmin in Figure 2.1 (left) quickly Results. The empirical primal dual gap L decreases toward the optimal gap of zero. The convergence is fast and monotonic, supporting the predicted behavior from our theory. The test-time performance in Figure 2.1 (middle) shows the safety constraint is always satisfied, while the main 27 Figure 2.1: FrozenLake Results. (Left) Empirical duality gap of algorithm 2 vs. optimal gap. (Middle) Comparison of returned policy and others w.r.t. (top) main objective value and (bottom) safety constraint value. (Right) FQE vs. other OPE methods on a standalone basis. objective cost also smoothly converges to the optimal value achieved by an online RL baseline (DQN) trained without regard to the constraint. The returned policy significantly outperformed the data gathering policy 𝜋D on both main and safety cost. Car Racing. Environment & Data Collection. The car racing environment (OpenAI), is a highdimensional domain where the state is a 96 ˆ 96 ˆ 3 tensor of raw pixels. The action space A “ tsteering ˆ gas ˆ brakeu takes 12 discretized values. The goal in this episodic task is to traverse over 95% of the track, measured by a given number of “tiles” as a proxy for distance coverage. The agent receives a reward (negative cost) for each unique tile crossed and no reward if the agent is off-track. A small positive cost applies at every time step, with maximum horizon of 1000 for each episode. With these costs given by the environment, one can train online RL agent using DDQN (Van Hasselt et al., 2016). We collect « 5000 trajectories from DDQN’s randomization, resulting in data set D with « 94000 transition tuples. Fast Driving under Behavioral Constraints. We study the problem of minimizing environment cost while subject to two behavioral constraints: smooth driving and lane centering. The first constraint 𝐺 0 approximates smooth driving by 𝑔0 p𝑥, 𝑎q “ 1 if 𝑎 contains braking action, and 0 otherwise. The second constraint cost 𝑔1 measures 28 Figure 2.2: CarRacing Results. (Left) & (Middle) (Lower is better) Comparing our algorithm, regularized LSPI, online RL w/o constraints, behavior policy 𝜋 𝐷 w.r.t. main cost objectives and two constraints. (Right) FQE vs. other OPE methods on a standalone basis. the distance between the agent and center of lane at each time step. This is a highly challenging setup since three objectives and constraints are in direct conflict with one another, e.g., fast driving encourages the agent to cut corners and apply frequent brakes to make turns. Outside of this work, we are not aware of previous work in policy learning with 2 or more constraints in high-dimensional settings. Baseline and Procedure. As a naïve baseline, DDQN achieves low cost, but exhibits “non-smooth” driving behavior (see our supplementary videos). We set the threshold for each constraint to 75% of the DDQN benchmark. We also compare against regularized batch RL algorithms (Farahmand et al., 2009), specifically regularized LSPI. We instantiate our subroutines, FQE and FQI, with multi-layered CNNs. We augment LSPI’s linear policy with non-linear features derived from a well-performing FQI model. Results. The returned mixture policy from our algorithm achieves low main objective cost, comparable with online RL policy trained without regard to constraints. After several initial iterations violating the braking constraint, the returned policy - corresponding to the appropriate 𝜆 trade-off - satisties both constraints, while improving the main objective. The improvement over data gathering policy is 29 significant for both constraints and main objective. Regularized policy learning is an alternative approach to (OPT) (section 2.2). We provide the regularized LSPI baseline the same set of 𝜆 found by our algorithm for one-shot regularized learning (Figures 2.2 (left & middle)). While regularized LSPI obtains good performance for the main objective, it does not achieve acceptable constraint satisfaction. By default, regularized policy learning requires parameter tuning heuristics. In principle, one can perform a grid-search over a range of parameters to find the right combination - we include such an example for both regularized LSPI and FQI in Appendix A.8. However, since our objective and constraints are in conflict, main objective and constraint satisfaction of policies returned by one-shot regularized learning are sensitive to step changes in 𝜆. In constrast, our approach is systematic, and is able to avoid the curse-of-dimensionality of brute-force search that comes with multiple constraints. In practice, one may wish to deterministically extract a single policy from the returned mixture for execution. A de-randomized policy can be obtained naturally from our algorithm by selecting the best policy from the existing FQE’s estimates of individual Best-response policies. Off-Policy Evaluation The off-policy evaluation by FQE is critical for updating policies in our algorithm, and is ultimately responsible for certifying constraint satisfaction. While other OPE methods can also be used in place of FQE, we find that the estimates from popular methods are not sufficiently accurate in a high-dimensional setting. As a standalone comparison, we select an individual policy and compare FQE against PDIS (Precup et al., 2000), DR (Jiang and Li, 2016a) and WDR (Thomas and Brunskill, 2016b) with respect to the constraint cost evaluation. To compare both accuracy and dataefficiency, for each domain we randomly sample different subsets of dataset D (from 10% to 100% transitions, 30 trials each). Figure 2.1 (right) and 2.2 (right) illustrate the difference in quality. In the FrozenLake domain, FQE performs competitively with the top baseline method (DR and WDR), converging to the true value estimate as the data subsample grows close to 100%. In the high-dimensional car domain, FQE signficantly outperforms other methods. 2.6 Other Related Work Constrained MDP (CMDP). Among the most important techniques for solving CMDP are the Lagrangian approach and solving the dual LP program via occupation 30 measure(Altman, 1999). However, these approaches require known MDP, and small state dimension so that solving via an LP is tractable. More recently, the constrained policy optimization approach (CPO) by (Achiam et al., 2017) learns a policy when the model is not initially known. The focus of CPO is on online safe exploration, and thus is not directly comparable to our setting. Other approaches (Cheng et al., 2019d; Dalal et al., 2018) address safe exploration by building the constraint directly into the policy. Multi-objective Reinforcement Learning (MORL). (Van Moffaert and Nowé, 2014; Roijers et al., 2013) Approaches to MORL have largely focused on approximating the Pareto frontier that trades-off competing objectives (Van Moffaert and Nowé, 2014; Roijers et al., 2013). The underlying approach to MORL frequently relies on linear or non-linear scalarization of rewards to heuristically turns the problem into a standard RL problem. Our proposed approach represents another systematic paradigm to solve MORL, whether in batch or online settings. 2.7 Discussion We have presented in this chapter a systematic approach for batch policy learning under multiple constraints. Our problem formulation can accommodate general definition of constraints, as partly illustrated by our experiments. We provide guarantees for our algorithm for both the main objective and constraint satisfaction. Our empirical results show a promise of making constrained batch policy learning applicable for real-world domains, where behavior data is abundant. While our algorithmic development focuses on the batch setting, i.e.,our implementation complies with the steps laid out in Algorithm 2. Adopting the proposed algorithmic approach to the online setting is relatively straightforward. However, in very large scale or high-dimensional problems, we may consider a noisy update version for both policy learning and evaluation. We leave the theorerical and practical exploration of this extension to future work. In our high-dimensional domain with long horizon, our proposed FQE algorithm for OPE achieves strong results. More extensive comparisons between FQE and other contemporary OPE methods deserve further study. We will focus on the problem of off-policy value estimation in the next chapter. 31 Chapter 3 OFF-POLICY VALUE ESTIMATION FOR REINFORCEMENT LEARNING Summary. As seen from the previous chapter, the problem of off-policy value estimation (OPE) is critical for reliable value-based approaches to policy learning. Also known as off-policy policy evaluation in reinforcement learning context, OPE has recently attracted a significant attention due to its practical impoirtance. However, the disparate experimental conditions in recent OPE literature make it difficult both for practitioners to choose a reliable estimator for their application domain, as well as for researchers to identify fruitful research directions. In this chapter, we present the first detailed empirical study of a broad suite of OPE methods. Based on thousands of experiments and empirical analysis, we offer a summarized set of guidelines to advance the understanding of OPE performance in practice, and suggest directions for future research. Along the way, our empirical findings challenge several commonly held beliefs about which class of approaches tends to perform well. Our accompanying software implementation serves as a first comprehensive benchmark for OPE. 3.1 Introduction to Off-Policy Value Estimation We focus on understanding the relative performance of existing methods for offpolicy value estimation (OPE), which is the problem of estimating the value of a target policy using only pre-collected historical data generated by another policy. The earliest OPE methods rely on classical importance sampling to handle the distribution mismatch between the target and behavior policies (Precup et al., 2000). Many advanced OPE methods have since been proposed for both contextual bandits (Dudík et al., 2011b; Bottou et al., 2013; Swaminathan et al., 2017; Wang et al., 2017; Li et al., 2015; Ma et al., 2020) and reinforcement learning settings (Jiang and Li, 2016a; Dudík et al., 2011b; Farajtabar et al., 2018a; Liu et al., 2018c; Xie et al., 2019). These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations (Li et al., 2011; Wiering, 2000; Bottou et al., 2013; Bang and Robins, 2005), and the increasing importance of off-policy learning and counterfactual reasoning more broadly (Degris et al., 2012; Thomas et al., 2017; Munos et al., 2016; Le et al., 2019b; Liu et al., 2019; Nie et al., 32 2019). OPE is also closely related to the problem of dynamic treatment regimes in the causal inference literature (Murphy et al., 2001). Empirical validations have long contributed to the scientific understanding and advancement of machine learning techniques (Chapelle and Li, 2011; Caruana et al., 2008; Caruana and Niculescu-Mizil, 2006). Recently, many have called for careful examination of empirical findings of contemporary deep learning and deep reinforcement learning efforts (Henderson et al., 2018; Locatello et al., 2019). As OPE is central to real-world applications of reinforcement learning, an in-depth empirical understanding is critical to ensure usefulness and accelerate progress. While many recent methods are built on sound mathematical principles, a practitioner is often faced with a non-trivial task of selecting the most appropriate estimator for their application. A notable gap in current literature is a comprehensive empirical understanding of contemporary methods, due in part to the disparate testing environments and varying experimental conditions among prior work. Consequently, there is little holistic insight into where different methods particularly shine, nor a systematic summary of the challenges one may encounter when in different scenarios. Researchers and practitioners may reasonably deduce the following commonly held impressions from surveying the literature: 1. Doubly robust methods are often assumed to outperform direct and importance sampling methods. 2. Horizon length is the primary driver of poor performance for OPE estimators. 3. Model-based is the go-to direct method, either standalone or as part of a doubly-robust estimator. The reality, as we will discuss, is much more nuanced. In this chapter, we take a closer look at recently proposed methods and offer a thorough empirical study of a wide range of estimators. We design various experimental conditions to explore the success and failure modes of different methods. We synthesize general insights to guide practitioners, and suggest directions for future research. Finally, we provide a highly extensive software package that can interface with new experimental environments and methods to run new OPE experiments at scale. Notations and Preliminaries for the Chapter As per RL standard, we represent the environment by x𝑋, 𝐴, 𝑃, 𝑅, 𝛾y. 𝑋 is the state space (or observation space in the non-Markov case), 𝐴 is the (finite) action space, IPS Methods 33 Standard IS Standard WIS Per-Decision IS Per-Decision WIS H MAGIC Doubly-Robust Weighted Doubly Robust r yb M id s od h t e More Robust Doubly-Robust Direct Q Regression Infinite Horizon Retrace(-) Tree Backup(-) Q/ (-) Fitted Q Evaluation Model Fitting Direct Methods Figure 3.1: Categorization of OPE methods. Some methods are direct but have IPS influence and thus fit slightly away from the direct methods axis. 𝑃 : 𝑋 ˆ 𝐴 ˆ 𝑋 Ñ r0, 1s is the transition function, 𝑅 : 𝑋 ˆ 𝐴 Ñ R is the reward, and discount factor 𝛾 P p0, 1s. A policy 𝜋 maps states to a distribution over actions, and 𝜋p𝑎|𝑥q denotes the probability of choosing 𝑎 P 𝐴 in 𝑥 P 𝑋. OPE is typically considered in the episodic RL setting. A behavior policy 𝜋 𝑏 generates a historical data set, 𝐷 “ t𝜏𝑖 u𝑖𝑁“1 , of 𝑁 trajectories (or episodes), where 𝑖 indexes over trajectories, and 𝜏𝑖 “ p𝑥 0𝑖 , 𝑎𝑖0 , 𝑟 0𝑖 , . . . , 𝑥𝑇𝑖 ´1 , 𝑎𝑇𝑖 ´1 , 𝑟𝑇𝑖 ´1 q. The episode length 𝑇 is frequently assumed to be fixed for notational convenience. In practice, one can pad additional absorbing states to handle variable lengths. Given a desired evaluation policy 𝜋 𝑒 , the OPE problem is to estimate the value 𝑉p𝜋 𝑒 q, defined as: « ff 𝑇ÿ ´1 𝑉p𝜋 𝑒 q “ E𝑥 „𝑑0 𝛾 𝑡 𝑟 𝑡 |𝑥 0 “ 𝑥 , 𝑡 “0 with 𝑎 𝑡 „ 𝜋 𝑒 p¨|𝑥𝑡 q, 𝑥𝑡 `1 „ 𝑃p¨|𝑥𝑡 , 𝑎 𝑡 q, 𝑟 𝑡 „ 𝑅p𝑥𝑡 , 𝑎 𝑡 q, and 𝑑0 is the initial state distribution. 3.2 Overview of Off-Policy Value Estimators OPE methods were historically categorized into importance sampling, direct, and doubly robust methods. This demarcation was first introduced for contextual bandits (Dudík et al., 2011b), and later to reinforcement learning (Jiang and Li, 2016a). Some recent methods have blurred the boundary of these categories, such 34 as Retrace(𝜆) (Munos et al., 2016) that uses a product of importance weights of multiple time steps for off-policy 𝑄 correction, and MAGIC (Thomas and Brunskill, 2016a) that switches between importance weighting and direct methods. In this section, we propose to regroup OPE into three similar classes of methods, but with expanded definition for each category. Figure 3.1 provides an overview of OPE methods that we consider. The relative positioning of different methods reflects how close they are to being a pure regression-based estimator versus a pure importance sampling-based estimator. Appendix B.4 contains a full description of all methods under consideration. Inverse Propensity Scoring (IPS) Inverse Propensity Scoring (IPS), also called importance sampling, is widely used in statistics (Powell and Swann, 1966; Hammersley and Handscomb, 1964; Horvitz and Thompson, 1952) and RL (Precup et al., 2000). The key idea is to reweight the rewards in the historical data by the importance sampling ratio between 𝜋 𝑒 and 𝜋 𝑏 , i.e., how likely a reward is under 𝜋 𝑒 versus 𝜋 𝑏 . IPS methods yield consistent and (typically) unbiased estimates; however the product of importance weights can be unstable for long time horizons. The cumulative importance weight between śminp 𝑗 1 ,𝑇 ´1q 𝜋𝑒 p𝑎𝑖𝑡 |𝑥𝑡𝑖 q 𝑖 1 (where 𝜌𝑡:𝑡 𝜋 𝑒 and 𝜋 𝑏 is written as 𝜌𝑖𝑗: 𝑗 1 “ 𝑡 “ 𝑗 1 “ 1 for 𝑡 ă 𝑡). 𝜋 𝑏 p𝑎 𝑖𝑡 |𝑥 𝑡𝑖 q ř Weighted IPS replaces a normalization factor 𝑁 by 𝑤 𝑗: 𝑗 1 “ 𝑁1 𝑖𝑁“1 𝜌𝑖𝑗: 𝑗 1 . The weighted versions are biased but strongly consistent. ř ř 𝜌𝑖 Importance Sampling (IS) takes the form: 𝑖𝑁“1 0:𝑇𝑁´1 𝑇𝑡 “´01 𝛾 𝑡 𝑟 𝑡 . There are three other main IPS variants that we consider: Per-Decision Importance Sampling (PDIS), Weighted Importance Sampling (WIS) and Per-Decision WIS (PDWIS) (see Appendix Table B.14 for full definitions). Other variants of IPS exist but are neither consistent nor unbiased (Thomas, 2015). IPS often assumes known 𝜋 𝑏 , which may not be possible – one approach is to estimate 𝜋 𝑏 from data (Hanna et al., 2019), resulting in a potentially biased estimator that can sometimes outperform traditional IPS methods. Direct Methods (DM) The main distinction of direct methods from IPS is the focus on regression-based techniques to (more) directly estimate the value functions of the evaluation policy (𝑄 𝜋𝑒 or 𝑉 𝜋𝑒 ). We consider eight different direct approaches, described completely in appendix B.4. Similar to policy learning literature, we can view OPE through the 35 Table 3.1: Environment parameters - Part 1 Environment Graph Graph-MC MC Pix-MC Enduro Markov? State/Obs 𝑇 Stoch Env? Stoch Rew? Sparse Rew? 𝑄ˆ Func. Class yes position 4 or 16 variable variable variable tabular yes position 250 no no terminal tabular yes [pos, vel] 250 no no terminal linear/NN yes pixels 250 no no terminal NN yes pixels 1000 no no dense NN Table 3.2: Environment parameters - Part 2 Environment Graph-POMDP GW Pix-GW Markov? State/Obs 𝑇 Stoch Env? Stoch Rew? Sparse Rew? 𝑄ˆ Func. Class no position 2 or 8 no no terminal tabular yes position 25 no no dense tabular yes pixels 25 variable no dense NN lens of model-based vs. model-free approaches1. Model-based. Perhaps the most commonly used DM is model-based (also called approximate model, denoted AM), where the transition dynamics, reward function and termination condition are directly estimated from historical data (Jiang and Li, 2016a; Paduraru, 2013). The resulting learned MDP is then used to compute the value of 𝜋 𝑒 , e.g., by Monte-Carlo policy evaluation. p 𝜃q, parameterized by 𝜃, is Model-free. Estimating the action-value function 𝑄p¨; p 𝑒q “ the focus of several model-free approaches. The value estimate is then: 𝑉p𝜋 ř ř 𝑁 1 𝑖 p 𝑖 𝑖 “1 𝑎 P 𝐴 𝜋 𝑒 p𝑎|𝑠0 q𝑄p𝑥 0 , 𝑎; 𝜃q. A simple example is Fitted Q Evaluation (FQE) 𝑁 (Le et al., 2019a), which is a model-free counterpart to AM, and is functionally a policy evaluation counterpart to batch Q learning. FQE learns a sequence of 1 the distinction is arguably blurry. We stick with this convention simply for linguistic convenience 36 p 𝜃q “ lim 𝑘 Ñ8 𝑄 p 𝑘 , where: estimators 𝑄p¨, 𝑁 𝑇 ´1 1 ÿÿ p p 𝑄 𝑘 “ min p𝑄 𝑘 ´1 p𝑥𝑡𝑖 , 𝑎𝑖𝑡 ; 𝜃q ´ 𝑦𝑖𝑡 q2 , 𝜃 𝑁 𝑖 “1 𝑡 “0 p 𝑘 ´1 p𝑥 𝑖 , ¨; 𝜃q, 𝑦𝑖𝑡 ” 𝑟 𝑡𝑖 ` 𝛾E𝜋𝑒 𝑄 𝑡 `1 p0 ” 0. 𝑄 Indeed, several model-free methods originated from off-policy learning settings, but are also natural for OPE. 𝑄 𝜋 p𝜆q (Harutyunyan et al., 2016) can be viewed as a generalization of FQE that looks to the horizon limit to incorporate the long-term value into the backup step. Retrace(𝜆) (Munos et al., 2016) and Tree-Backup(𝜆) (Precup et al., 2000) also use full trajectories, but additionally incorporate varying levels of clipped importance weights adjustment. The 𝜆-dependent term mitigates instability in the backup step, and is chosen based on experimental findings of Munos et al. (2016). Q Regression (Q-Reg) and More Robust Doubly-Robust (MRDR) (Farajtabar et al., 2018a) are two recently proposed direct methods that make use of cumulative importance weights in deriving the regression estimate for 𝑄 𝜋𝑒 , solved through a quadratic program. MRDR changes the objective of the regression to that of directly minimizing the variance of the Doubly-Robust estimator (see Section 3.2). Liu et al. (2018c) recently proposed a method for the infinite horizon setting (IH). While IH can be viewed as a Rao-Blackwellization of the IS estimator, we include it in the DM category because it essentially solves the Bellman equation for state distributions and requires function approximation, which are more characteristic of DM. IH shifts the focus from importance sampling over action sequences to estimating the importance ratio 𝜔 between state density distributions induced by 𝜋 𝑏 and 𝜋 𝑒 . This ratio replaces all but the final importance weights 𝜌𝑇 ´1 in the IH estimate, which resembles IS. More recently, several estimators inspired by density ratio estimation idea have been proposed (Nachum et al., 2019; Uehara and Jiang, 2019; Xie et al., 2019) - we will leave evaluation of these new extensions for future work. Hybrid Methods (HM) Hybrid methods subsume doubly robust-like approaches, which combine aspects of both IPS and DM. Standard doubly robust OPE (denoted DR) (Jiang and Li, 2016a) is an unbiased estimator that leverages a DM to decrease the variance of the 37 unbiased estimates produced by importance sampling techniques: 𝑁 p 𝑖 ÿ 𝑉p𝑥 0 q 𝑁 𝑖 “1 𝑁 𝑇 ´1 1 ÿÿ 𝑡 𝑖 𝑖 p 𝑖 𝑖 p 𝑖 qs. ` 𝛾 𝜌0:𝑡 r𝑟 𝑡 ´ 𝑄p𝑥𝑡 , 𝑎 𝑡 q ` 𝛾𝑉p𝑥 𝑡 `1 𝑁 𝑖 “1 𝑡 “0 Other HMs include Weighted Doubly-Robust (WDR) and MAGIC (see Appendix B.4). WDR self-normalizes the importance weights (similar to WIS). MAGIC introduces adaptive switching between DR and DM; in particular, one can imagine using DR to estimate the value for part of a trajectory and then using DM for the remainder. Using this idea, MAGIC (Thomas and Brunskill, 2016a) finds an optimal linear combination among a set that varies the switch point between WDR and DM. p𝜋𝑒 p𝑥, 𝑎; 𝜃q yields a set of corresponding DR, WDR, Note that any DM that returns 𝑄 and MAGIC estimators. As a result, we consider twenty-one hybrid approaches in our experiments. 3.3 Experiments Experiment Design Principles. We consider several domain characteristics (simplecomplex, deterministic-stochastic, sparse-dense rewards, short-long horizon), 𝜋 𝑏 , 𝜋 𝑒 pairs (close-far), and data sizes 𝑁 (small-large), to study OPE performance under varying conditions. We use two standard RL benchmarks from OpenAI Brockman et al. (2016): Mountain Car (MC) and Enduro Atari game. As many RL benchmarks are fixed and deterministic, we design 6 additional environments that allow control over various conditions: (i) Graph domain (tabular, varying stochasticity and horizon), (ii) Graph-POMDP (tabular, control for representation), (iii) Graph-MC (simplifying MC to tabular case), (iv) Pixel-MC (study MC in high-dimensional setting), (v) Gridworld (tabular, long horizon version) and (vi) Pixel-Gridworld (controlled Gridworld experiments with function approximation). All together, our benchmark consists of eight environments with characteristics summarized in Table 3.1. Complete descriptions can be found in Appendix B.5. Protocol & Metrics. Each experiment depends on specifying environment and its properties, behavior policy 𝜋 𝑏 , evaluation policy 𝜋 𝑒 , and number of trajectories 𝑁 from rolling-out 𝜋 𝑏 for historical data. The true on-policy value 𝑉p𝜋 𝑒 q is the MonteCarlo estimate via 10, 000 rollouts of 𝜋 𝑒 . We repeat each experiment 𝑚 “ 10 times with different random seeds. We judge the quality of a method via two metrics: 38 1 ř𝑚 ř p𝑉pp𝜋𝑒 q𝑖 ´ 𝑚 𝑗 “1 𝑉 p𝜋𝑒 q 𝑗 q2 ř • Relative mean squared error (Relative MSE): 𝑚1 𝑖𝑚“1 , 2 p 𝑚1 𝑚 𝑗 “1 𝑉 p 𝜋 𝑒 q 𝑗 q which allows a fair comparison across different conditions.2 • Near-top Frequency: For each experimental condition, we include the number of times each OPE estimator is within 10% of the best performing estimator to facilitate aggregate comparison across domains. Implementation & Hyperparameters. With thirty-three different OPE methods considered, we run thousands of experiments across the above eight domains. Hyperparameters are selected based on publication, code release or author consultation. We maintain a consistent set of hyperparameters for each estimator and each environment across experimental conditions (see hyperparameter choice in appendix Table B.23). We create a software package that allows running experiments at scale and easy integration with new domains and techniques for future research. Due to limited space, we will show the results from selected experiment conditions. The complete results, with highlighted best method in each class, are available in the appendix. 3.4 Results What is the best method? The first important takeaway is that there is no clear-cut winner: no single method or method class is consistently the best performer, as multiple environmental factors can influence the accuracy of each estimator. With that caveat in mind, based on the aggregate top performance metrics, we can recommend the following estimators for each method class (See Table 3.3 and appendix Table B.2). Inverse propensity scoring (IPS). In practice, weighted importance sampling, which is biased, tends to be more accurate and data-efficient than unbiased basic importance sampling methods. Among the four IPS-based estimators, PDWIS tends to perform best (Figure 3.4 left). Direct methods (DM). Generally, FQE, 𝑄 𝜋 p𝜆q, and IH tend to perform the best among DM (appendix Table B.2). FQE tends to be more data efficient and is the best method when data is limited (Figure 3.5). 𝑄 𝜋 p𝜆q generalizes FQE to multistep backup, and works particularly well with more data, but is computationally expensive in complex domains. IH is highly competitive in long horizons and with high policy mismatch in a tabular setting (appendix Tables B.6, B.7). In pixel-based 2 The performance metric in prior OPE work is typically mean squared error MSE“ 1 ř𝑚 2 p 𝑖 “1 p𝑉p𝜋 𝑒 q𝑖 ´ 𝑉p𝜋 𝑒 q𝑖 q 𝑚 39 Table 3.3: Method Selection Guidelines - Part 1 Class Recommendation When to use Direct FQE 𝑄p𝜆q IH PDWIS MAGIC FQE MAGIC 𝑄p𝜆q Stochastic env, severe policy mismatch Compute non-issue, moderate policy mismatch Long horizon, mild policy mismatch, good kernel Short horizon, mild policy mismatch Severe model misspecification Compute non-issue, severe model misspecification IPS Hybrid domains, however, choosing a good kernel function for IH is not straightforward, and IH can underperform other DM (appendix Table B.10). We provide a numerical comparison among direct methods for tabular (appendix Figure B.11) and complex settings (Figure 3.4 center). Table 3.4: Method Selection Guidelines - Part 2. (For Near-top Frequency, see definition in Section 3.3 and support in Table B.2) Class Recommendation Prototypical env. Direct FQE 𝑄p𝜆q IH PDWIS MAGIC FQE MAGIC 𝑄p𝜆q Graph, MC, Pix-MC GW/Pix-GW Graph-MC Graph Graph-POMDP, Enduro Graph-POMDP IPS Hybrid Near-top Freq. 23.7% 15.0% 19.0% 4.7% 30.0% 17.3% Hybrid methods (HM). With the exception of IH, each DM corresponds to three HM: standard doubly robust (DR), weighted doubly robust (WDR), and MAGIC. For each DM, its WDR version often outperforms its DR version. MAGIC can often outperform WDR and DR. However, MAGIC comes with additional hyperparameters, as one needs to specify the set of partial trajectory length to be considered. Unsurprisingly, their performance highly depends on the underlying DM. In our experiments, FQE and 𝑄 𝜋 p𝜆q are typically the most reliable: MAGIC with FQE or MAGIC with 𝑄 𝜋 p𝜆q tend to be among the best hybrid methods (see appendix Figures B.17 - B.21). Key drivers of method accuracy The main reason for the inconsistent performance of estimators is various environmental factors that are inadequately studied from prior work. These coupled factors 40 often impact accuracy interdependently: • Representation mismatch: Function approximators with insufficient representation power weaken DM, and so do overly rich ones as they cause overfitting (e.g., tabular classes). These issues do not impact IPS. Severe misspecification favors HM and weakens DM. • Horizon length: Long horizons hurt all methods, but especially those dependent on importance weights (including IPS, HM and some DM). • Policy mismatch: Large divergence between 𝜋 𝑏 and 𝜋 𝑒 hurts all methods, but tends to favor DM in the small data regime relative to HM and IPS. HM will catch up with DM as data size increases. • Bad estimation of unknown behavior policy:3 𝜋 𝑏 estimation quality depends on the state and action dimensionality, and historical data size. Poor 𝜋 𝑏 estimates cause HM and IPS to underperform simple DM. • Environment / Reward stochasticity: Stochastic environments hurt the data efficiency of all methods, but favor DM over HM and IPS. We perform a series of controlled experiments to isolate the impact of these factors. Figure 3.3 shows a typical comparison of the best performing method in each class, under a tabular setting with both short and long horizons, and a large mismatch between 𝜋 𝑏 and 𝜋 𝑒 . The particular best method in each class may change depending on the specific conditions. Within each class, a general guideline for method selection is summarized in Table 3.3. The appendix contains the full empirical results of all experiments. A recipe for method selection Figure 3.2 summarizes our general guideline for navigating key factors that affect the accuracy of different estimators. To guide the readers through the process, we now dive further into our experimental design to test various factors, and discuss the resulting insights. Do we potentially have representation mismatch? Representation mismatch comes from two sources: model misspecification and poor generalization. Model misspecification refers to the insufficient representation power of the function class used 3 Poor estimation of 𝜋 𝑏 can also be seen as model misspecification. We distinguish the representation issue of 𝜋 𝑏 from other representation issues related to DM 41 Figure 3.2: Method Class Selection Decision Tree. Numerical support can be found in Appendix B.2. to approximate either the transition dynamics (AM), value function (other DM), or state distribution density ratio (in IH). Tabular representation for MDP controls for representation mismatch by ensuring adequate function class capacity, as well as zero inherent Bellman error (left branch, Fig 3.2). In such case, we may still suffer from poor generalization without sufficient data coverage, which depends on other factors in the domain settings. The effect of representation mismatch (right branch, Fig 3.2) can be understood via two controlled scenarios: • Misspecified and poor generalization: We expose the impact of this severe mismatch scenario via the Graph POMDP construction, where selected information are omitted from an otherwise equivalent Graph MDP. HM substantially outperform DM in this setting (Figure 3.3 right versus left). • Misspecified but good generalization: Function class such as neural networks has powerful generalization ability, but may introduce bias and inherent Bellman error4 (Munos and Szepesvári, 2008; Chen and Jiang, 2019) (see linear vs. neural networks comparison for Mountain Car in appendix Fig B.8). Still, powerful function approximation makes (biased) DM very competitive 4 defined as sup 𝑔PF inf 𝑓 PF || 𝑓 ´ T 𝜋 𝑔|| 𝑑 𝜋 , where F is function class chosen for approximation, and 𝑑 𝜋 is state distribution induced by evaluation policy 𝜋 42 Figure 3.3: Comparing IPS versus DM versus HM under short and long horizon, large policy mismatch and large data. Left: (Graph domain) Deterministic environment. Center: (Graph domain) Stochastic environment and rewards. Right: (Graph-POMDP) Model misspecification (POMDP). Minimum error per class is shown. with HM, especially under limited data and in complex domains (see pixelGridworld in appendix Fig B.22-B.24). However, function approximation bias may cause serious problem for high dimensional and long horizon settings. In the extreme case of Enduro (very long horizon and sparse rewards), all DM fail to convincingly outperform a naïve average of behavior data (appendix Fig B.7). Short horizon vs. Long horizon? It is well-known that IPS methods are sensitive to trajectory length (Li et al., 2015). Long horizon leads to an exponential blowup of the importance sampling term, and is exacerbated by significant mismatch between 𝜋 𝑏 and 𝜋 𝑒 . This issue is inevitable for any unbiased estimator (Jiang and Li, 2016b) (a.k.a., the curse of horizon (Liu et al., 2018b)). Similar to IPS, DM relying on importance weights also suffer from long horizon (appendix Fig B.11), though to a lesser degree. IH aims to bypass the effect of cumulative weighting in long horizons, and indeed performs substantially better than IPS methods in very long horizon domains (Fig 3.4 left). A frequently ignored aspect in previous OPE work is a proper distinction between fixed, finite horizon tasks (IPS focus), infinite horizon tasks (IH focus), and indefinite horizon tasks, where the trajectory length is finite but varies depending on the policy. Many applications should properly belong to the indefinite horizon category.5 Applying HM in this setting requires proper padding of the rewards (without altering the value function in the infinite horizon limit) as DR correction typically assumes fixed length trajectories. 5 Applying IH in the indefinite horizon case requires setting up an absorbing state that loops over itself with zero terminal reward. 43 How different are behavior and target policies? Similar to IPS, the performance of DM is negatively correlated with the degree of policy mismatch. Figure 3.5 shows the interplay of increasing policy mismatch and historical data size, on the top DM 𝜋 p𝑎 |𝑥 q in the deterministic gridworld. We use psup𝑎P 𝐴,𝑥 P 𝑋 𝜋𝑏𝑒 p𝑎|𝑥 q q𝑇 as an environmentindependent metric of mismatch between the two policies. The performance of the top DM (FQE, 𝑄 𝜋 p𝜆q, IH) tend to hold up better than IPS methods when the policy gap increases (appendix Figure B.13). FQE and IH are best in the small data regime, and 𝑄 𝜋 p𝜆q performs better as data size increases (Figure 3.5). Increased policy mismatch weakens the DM that use importance weights (Q-Reg, MRDR, Retrace(𝜆) and Tree-Backup(𝜆)). Do we have a good estimate of the behavior policy? Often the behavior policy may not be known exactly and requires estimation, which can introduce bias and cause HM to underperform DM, especially in low data regime (e.g., pixel gridworld appendix Figure B.22-B.24). Similar phenomenon was observed in the statistics literature (Kang et al., 2007). As the data size increases, HMs regain the advantage as the quality of the 𝜋 𝑏 estimate improves. Is the environment stochastic or deterministic? While stochasticity affects all methods by straining the data requirement, HM are more negatively impacted than DM (Figure 3.3 center, Figure B.12). This can be justified by e.g., the variance analysis of DR, which shows that the variance of the value function with respect to stochastic transitions will be amplified by cumulative importance weights and then contribute to the overall variance of the estimator; see Jiang and Li (2016b, Theorem 1) for further details. We empirically observe that DM frequently outperform their DR versions in the small data case (Figure B.12). In a stochastic environment and tabular setting, HM do not provide significant edge over DM, even in short horizon case. The gap closes as the data size increases (Figure B.12). Challenging common wisdom We close this section by briefly revisiting commonly held beliefs about high-level performance of OPE methods. Are HM always better than DM? No. Overall, DM are surprisingly competitive with HM. Under high-dimensionality, long horizons, estimated behavior policies, or reward/environment stochasticity, HM can underperform simple DM, sometimes significantly (e.g., see appendix Figure B.12). Concretely, HM can perform worse than DM in the following scenarios that we 44 Figure 3.4: Left: (Graph domain) Comparing IPS (and IH) under short and long horizon. Mild policy mismatch setting. PDWIS is often best among IPS. But IH outperforms in long horizon. Center: (Pixel-MC) Comparing direct methods in high-dimensional, long horizon setting. Relatively large policy mismatch. FQE and IH tend to outperform. AM is significantly worse in complex domains. Retrace(𝜆), Q(𝜆) and Tree-Backup(𝜆) are very computationally expensive and thus excluded. Right: (Pixel Gridworld) Comparing MAGIC with different base DM and different data size. Large policy mismatch, deterministic environment, known 𝜋 𝑏 . tested: • Tabular with large policy mismatch, or stochastic environments (appendix Figure B.12, Table B.4, B.7). • Complex domains with long horizon and unknown behavior policy (appendix Figure B.22-B.24, Table B.9). When data is sufficient, or model misspecification is severe, HM do provide consistent improvement over DM. Is horizon length the most important factor? No. Despite conventional wisdom suggesting IPS methods are most sensitive to horizon length, we find that this is not 𝜋 p 𝑎 |𝑥 q always the case. Policy divergence sup𝑎P 𝐴,𝑥 P 𝑋 𝜋𝑏𝑒 p𝑎|𝑥 q can be just as, if not more, meaningful. For comparison, we designed two scenarios with identical mismatch 𝜋 p𝑎 |𝑥 q psup𝑎P 𝐴,𝑥 P 𝑋 𝜋𝑏𝑒 p𝑎|𝑥 q q𝑇 as defined in Section 3.4 (see appendix Tables B.12, B.13). Starting from a baseline scenario of short horizon and small policy divergence (appendix Table B.11), extending horizon length leads to 10ˆ degradation in accuracy, while a comparable increase in policy divergence causes a 100ˆ degradation. How good is model-based direct method (AM)? AM can be among the worst performing direct methods (appendix Table B.2). While AM performs well in tabular setting in the large data case (appendix Figure B.11), it tends to perform poorly in high dimensional settings with function approximation (e.g., Figure 3.4 center). Fitting the transition model 𝑃p𝑥 1 |𝑥, 𝑎q is often more prone to small errors than directly approximating 𝑄p𝑥, 𝑎q. Model fitting errors also compound with long horizons. 45 Figure 3.5: (Gridworld domain) Errors are directly correlated with policy mismatch but inversely correlated with data size. We pick the best direct methods for illustration. The two plots represent the same figure from two different vantage points. See full figures in appendix. Other Considerations Hypeparameter selection. As with many machine learning techniques, hyperparameter choice affects the performance of most estimators (except IPS estimators). The situation is more acute for OPE than the online off-policy learning setting, due to the lack of proper validation signal (such as online game score). When using function approximation, direct methods may not have satisfactory convergence, and require setting a reasonable termination threshold hyperparameter. Q-Reg and MRDR require extra care to avoid ill-conditioning, such as tuning with L1 and L2 regularization.6 Similarly, the various choice of the kernel function for IH and the index set for hybrid method such as MAGIC have large impact on the performance. In general, given the choice among different hybrid (or direct) methods, we recommend opting for simplicity as a guiding principle. Computational considerations. DM are generally significantly more computationally demanding than IPS. In complex domains, model-free iterative methods can be expensive in training time. Iterative DM that incorporate rollouts until the end of trajectories during training (Retrace(𝜆), 𝑄 𝜋 p𝜆q, Tree-Backup(𝜆)) are the most computationally demanding7, requiring an order of 𝑇 times the number of 𝑄ˆ 𝑘 ´1 p𝑥, 𝑎q lookups per gradient step compared to FQE. Model-based method (AM) are expensive at test time when coupled with HM, since rolling-out the learned model is required at every state along the trajectory.8 HM versions of direct methods require 𝑇 times more inference steps, which is often fast after training. In difficult 6 From correspondence with the authors. 7 (Munos et al., 2016) limits the rolling-out horizon to 16 in Atari domains, but for the policy learning scenario. 8 Unlike iterative DM (e.g., FQE), model-based method AM does not benefit from stochastic gradient speedup. Parallelizing the rollouts of AM is highly recommended. 46 tasks such as Atari games, running AM, Retrace(𝜆), 𝑄 𝜋 p𝜆q, Tree-Backup(𝜆) can be prohibitively expensive. Q-Reg, MRDR are non-iterative methods and thus are the fastest to execute among DM. The run-time of IH is dependent on the batch size in building a kernel matrix to compute state similarity. The batch size for IH should be as large as possible, but could significantly slow the training. Sparsity (non-smoothness) of the rewards: Methods that are dependent on cumulative importance weights are also sensitive to reward sparsity (Figure B.14). We recommend normalizing the rewards. As a rough guideline, zero-centering rewards often improve performance of methods that depend on importance weights. This seemingly naïve practice can be actually viewed as a special case of DR using a constant DM component (baseline), and can yield improvements over vanilla IPS (Jiang and Li, 2016a). 3.5 Discussion and Future Directions The most difficult environments break all estimators. Atari games pose significant challenges for contemporary techniques due to long horizon and high state dimensionality. It is possible that substantially more historical data is required for current OPE methods to succeed. However, to overcome computational challenge in complex RL domains, it is important to identify principled ways to stabilize iterative methods such as FQE, Retrace(𝜆), Q(𝜆) when using function approximation, as convergence is typically not attainable. Some recent progress has been made in stabilizing batch Q-learning in the off-policy learning setting (Fujimoto et al., 2019). It remains to be seen whether similar approach can also benefit DM for OPE. Lack of short-horizon benchmark in high-dimensional settings. Evaluation of other complex RL tasks with short horizon is currently beyond the scope of our study, due to the lack of a natural benchmark. We refer to prior work on OPE for contextual bandits, which are RL problems with horizon 1 (Dudík et al., 2011b). For contextual bandits, it has been shown that while DR is highly competitive, it is sometimes substantially outperformed by DM (Wang et al., 2017). New benchmark tasks should have longer horizon than contextual bandits, but shorter than typical Atari games. We also currently lack natural stochastic environments in high-dimensional RL benchmarks. An example candidate for medium horizon, complex OPE domain is NLP tasks such as dialogue. Other OPE settings. Below we outline several practically relevant settings that current literature has overlooked: 47 • Continuous actions. Recent literature on OPE has exclusively focused on finite actions. OPE for continuous action domains will benefit continuous control applications. Currently, continuous action domains will not work with all IPS and HM (see IPS for continuous contextual bandits by Kallus and Zhou (2018)). Among DM, perhaps only FQE may reasonable work with continuous action tasks with some adaptation. • Missing data coverage. A common assumption in the analysis of OPE is a full support assumption: 𝜋 𝑒 p𝑎|𝑥q ą 0 implies 𝜋 𝑏 p𝑎|𝑥q ą 0, which often ensure unbiasedness of estimators (Precup et al., 2000; Liu et al., 2018c; Dudík et al., 2011b). This assumption may not hold, and is often not verifiable in practice. Practically, violation of this assumption requires regularization of unbiased estimators to avoid ill-conditioning (Liu et al., 2018c; Farajtabar et al., 2018a). One avenue to investigate is to optimize bias-variance trade-off when the full support is not applicable. • Confounding variables. Existing OPE research often assumes that the behavior policy chooses actions solely based on the state. This assumption is often violated when the decisions in the historical data are made by humans instead of algorithms, who may base their decisions on variables not recorded in the data, causing confounding effects. Tackling this challenge, possibly using techniques from causal inference (Tennenholtz et al., 2019; Oberst and Sontag, 2019), is an important future direction. Evaluating new OPE estimators. More recently, several new OPE estimators have been proposed: (Nachum et al., 2019; Zhang et al., 2020) further build on the perspective of density ratio estimation from IH; Uehara and Jiang (2019) provides a closely related approach that learns value functions from important ratios; (Xie et al., 2019) proposes improvement over standard IPS by estimating marginalized state distribution in an analogous fashion to IH; (Kallus and Uehara, 2019a,b) analyze double reinforcement learning estimator that makes use of both estimates for 𝑄 function and state density ratio. While we have not included these new additions in our analysis, our software implementation is highly modular and can easily accommodate new estimators and environments. Algorithmic approach to method selection. While we have identified a general guideline for selecting OPE method, often it is not easy to judge whether some decision criteria are satisfied (e.g., quantifying model misspecification, degree of stochasticity, or appropriate data size). As more OPE methods continue to be 48 developed, an important missing piece is a systematic technique for model selection, given a high degree of variability among existing techniques. 49 Chapter 4 REGULARIZED LEARNING WITH POLICY-BASED STRUCTURE (SMOOTH IMITATION LEARNING) Summary. We study the problem of smooth imitation learning for online sequence prediction, where the goal is to train a policy that can smoothly imitate demonstrated behavior in a dynamic and continuous environment in response to online, sequential context input. Since the mapping from context to behavior is often complex, we take a learning reduction approach to reduce smooth imitation learning to a regression problem using complex function classes that are regularized to ensure smoothness. We present a learning meta-algorithm that achieves fast and stable convergence to a good policy. Our approach enjoys several attractive properties, including being fully deterministic, employing an adaptive learning rate that can provably yield larger policy improvements compared to previous approaches, and the ability to ensure stable convergence. Our empirical results demonstrate significant performance gains over previous approaches. 4.1 Introduction In many complex planning and control tasks, it can be very challenging to explicitly specify a good policy. For such tasks, the use of machine learning to automatically learn a good policy from observed expert behavior, also known as imitation learning or learning from demonstrations, has proven tremendously useful (Abbeel and Ng, 2004; Ratliff et al., 2009; Argall et al., 2009; Ross and Bagnell, 2010; Ross et al., 2011a; Jain et al., 2013). In this chapter, we study the problem of imitation learning for smooth online sequence prediction in a continuous regime. Online sequence prediction is the problem of making online decisions in response to exogenous input from the environment, and is a special case of reinforcement learning (see Section 4.2). We are further interested in policies that make smooth predictions in a continuous action space. Our motivating example is the problem of learning smooth policies for automated camera planning (Chen et al., 2016): determining where a camera should look given environment information (e.g., noisy person detections) and corresponding 50 demonstrations from a human expert.1 It is widely accepted that a smoothly moving camera is essential for generating aesthetic video (Gaddam et al., 2015). From a problem formulation standpoint, one key difference between smooth imitation learning and conventional imitation learning is the use of a “smooth” policy class (which we formalize in Section 4.2), and the goal now is to mimic expert demonstrations by choosing the best smooth policy. The conventional supervised learning approach to imitation learning is to train a classifier or regressor to predict the expert’s behavior given training data comprising input/output pairs of contexts and actions taken by the expert. However, the learned policy’s prediction affects (the distribution of) future states during the policy’s actual execution, and so violates the crucial i.i.d. assumption made by most statistical learning approaches. To address this issue, numerous learning reduction approaches have been proposed (Daumé III et al., 2009; Ross and Bagnell, 2010; Ross et al., 2011a), which iteratively modify the training distribution in various ways such that any supervised learning guarantees provably lift to the sequential imitation setting (potentially at the cost of statistical or computational efficiency). We present a learning reduction approach to smooth imitation learning for online sequence prediction, which we call SIMILE (Smooth IMItation LEarning). Building upon learning reductions that employ policy aggregation (Daumé III et al., 2009), we provably lift supervised learning guarantees to the smooth imitation setting and show much faster convergence behavior compared to previous work. Our contributions in this section can be summarized as: • We formalize the problem of smooth imitation learning for online sequence prediction, and introduce a family of smooth policy classes that is amenable to supervised learning reductions. • We present a principled learning reduction approach, which we call SIMILE. Our approach enjoys several attractive practical properties, including learning a fully deterministic stationary policy (as opposed to SEARN (Daumé III et al., 2009)), and not requiring data aggregation (as opposed to DAgger (Ross et al., 2011a)) which can lead to super-linear training time. 1 Access data at http://www.disneyresearch.com/publication/smooth-imitation- learning/ and code at http://github.com/hoangminhle/SIMILE. 51 • We provide performance guarantees that lift the the underlying supervised learning guarantees to the smooth imitation setting. Our guarantees hold in the agnostic setting, i.e., when the supervised learner might not achieve perfect prediction. • We show how to exploit a stability property of our smooth policy class to enable adaptive learning rates that yield provably much faster convergence compared to SEARN (Daumé III et al., 2009). • We empirically evaluate using the setting of smooth camera planning (Chen et al., 2016), and demonstrate the performance gains of our approach. 4.2 Formulating the Problem as Functional Regularization Let X “ t𝑥 1 , . . . , 𝑥𝑇 u Ă X𝑇 denote a context sequence from the environment X, and A “ t𝑎 1 , . . . , 𝑎𝑇 u Ă A𝑇 denote an action sequence from some action space A. Context sequence is exogenous, meaning 𝑎 𝑡 does not influence future context 𝑥𝑡 ` 𝑘 for 𝑘 ě 1. Let Π denote a policy class, where each 𝜋 P Π generates an action sequence A in response to a context sequence X. Assume X Ă R𝑚 , A Ă R 𝑘 are continuous and infinite, with A non-negative and bounded such that 0® ĺ 𝑎 ĺ 𝑅 1® @𝑎 P A. Predicting actions 𝑎 𝑡 may depend on recent contexts 𝑥𝑡 , . . . , 𝑥𝑡 ´ 𝑝 and actions 𝑎 𝑡 ´1 , . . . , 𝑎 𝑡 ´𝑞 . Without loss of generality, we define a state space S as t𝑠𝑡 “ r𝑥𝑡 , 𝑎 𝑡 ´1 su.2 Policies 𝜋 can thus be viewed as mapping states S “ X ˆ A to actions A. A roll-out of 𝜋 given context sequence X “ t𝑥 1 , . . . , 𝑥𝑇 u is the action sequence A “ t𝑎 1 , . . . , 𝑎𝑇 u: 𝑎 𝑡 “ 𝜋p𝑠𝑡 q “ 𝜋pr𝑥𝑡 , 𝑎 𝑡 ´1 sq, 𝑠𝑡 `1 “ r𝑥𝑡 `1 , 𝑎 𝑡 s @𝑡 P r1, . . . , 𝑇s . Note that unlike the general reinforcement learning problem, we consider the setting where the state space splits into external and internal components (by definition, 𝑎 𝑡 influences subsequent states 𝑠𝑡 ` 𝑘 , but not 𝑥𝑡 ` 𝑘 ). The use of exogenous contexts t𝑥𝑡 u models settings where a policy needs to take online, sequential actions based on external environmental inputs, e.g. smooth self-driving vehicles for obstacle avoidance, helicopter aerobatics in the presence of turbulence, or smart grid management for external energy demand. The technical motivation of this dichotomy is that we will enforce smoothness only on the internal state. 2 We can always concatenate consecutive contexts and actions. 52 Consider the example of autonomous camera planning for broadcasting a sport event (Chen et al., 2016). X can correspond to game information such as the locations of the players, the ball, etc., and A can correspond to the pan-tilt-zoom configuration of the broadcast camera. Manually specifying a good camera policy can be very challenging due to sheer complexity involved with mapping X to A. It is much more natural to train 𝜋 P Π to mimic observed expert demonstrations. For instance, Π can be the space of neural networks or tree-based ensembles (or both). Following the basic setup from (Ross et al., 2011a), for any policy 𝜋 P Π, let 𝑑𝑡𝜋 denote the distribution of states at time 𝑡 if 𝜋 is executed for the first 𝑡 ´ 1 time steps. ř Furthermore, let 𝑑 𝜋 “ 𝑇1 𝑇𝑡“1 𝑑𝑡𝜋 be the average distribution of states if we follow 𝜋 for all 𝑇 steps. The goal of imitation learning is to find a policy 𝜋ˆ P Π which minimizes the imitation loss under its own induced distribution of states: 𝜋ˆ “ arg min ℓ𝜋 p𝜋q “ arg min E𝑠„𝑑 𝜋 rℓp𝜋p𝑠qqs , 𝜋 PΠ (4.1) 𝜋 PΠ where the (convex) imitation loss ℓp𝜋p𝑠qq captures how well 𝜋 imitates expert demonstrations for state 𝑠. One common ℓ is squared loss between the policy’s decision and the expert demonstration: ℓp𝜋p𝑠qq “ }𝜋p𝑠q ´ 𝜋 ˚ p𝑠q}2 for some norm }.}. Note that computing ℓ typically requires having access to a training set of expert demonstrations 𝜋 ˚ on some set of context sequences. We also assume an agnostic setting, where the minimizer of (4.1) does not necessarily achieve 0 loss (i.e. it cannot perfectly imitate the expert). Smooth Imitation Learning & Smooth Policy Class In addition to accuracy, a key requirement of many continuous control and planning problems is smoothness (e.g., smooth camera trajectories). Generally, “smoothness" may reflect domain knowledge about stability properties or approximate equilibria of a dynamical system. We thus formalize the problem of smooth imitation learning as minimizing (4.1) over a smooth policy class Π. Most previous work on learning smooth policies focused on simple policy classes such as linear models (Abbeel and Ng, 2004), which can be overly restrictive. We instead define a much more general smooth policy class Π as a regularized space of complex models. Definition 4.2.1 (Smooth policy class Π). Given a complex model class F and a class of smooth regularizers H , we define smooth policy class Π Ă F ˆ H as satisfying: 53 Π fi t𝜋 “ p 𝑓 , ℎq, 𝑓 P F , ℎ P H | 𝜋p𝑠q is close to both 𝑓 p𝑥, 𝑎q and ℎp𝑎q @ induced state 𝑠 “ r𝑥, 𝑎s P Su where closeness is controlled by regularization. For instance, F can be the space of neural networks or decision trees and H be the space of smooth analytic functions. Π can thus be viewed as policies that predict close to some 𝑓 P F but are regularized to be close to some ℎ P H . For sufficiently expressive F , we often have that Π Ă F . Thus optimizing over Π can be viewed as constrained optimization over F (by H ), which can be challenging. Our SIMILE approach integrates alternating optimization (between F and H ) into the learning reduction. We provide two concrete examples of Π below. Example 4.2.1 (Π𝜆 ). Let F be any complex supervised model class, and define the simplest possible H fi tℎp𝑎q “ 𝑎u. Given 𝑓 P F , the prediction of a policy 𝜋 can be viewed as regularized optimization over the action space to ensure closeness of 𝜋 to both 𝑓 and ℎ: › ›2 › ›2 𝜋p𝑥, 𝑎q “ arg min › 𝑓 p𝑥, 𝑎q ´ 𝑎 1 › ` 𝜆 › ℎp𝑎q ´ 𝑎 1 › 𝑎 1 PA 𝑓 p𝑥, 𝑎q ` 𝜆𝑎 𝑓 p𝑥, 𝑎q ` 𝜆ℎp𝑎q “ , (4.2) 1`𝜆 1`𝜆 where regularization parameter 𝜆 trades-off closeness to 𝑓 and to previous action. For large 𝜆, 𝜋p𝑥, 𝑎q is encouraged make predictions that stays close to previous action 𝑎. “ Example 4.2.2 (Linear auto-regressor smooth regularizers). Let F be any complex supervised model class, and define H using linear auto-regressors, H fi tℎp𝑎q “ 𝜃 J 𝑎u, which model actions as a linear dynamical system (Wold, 1939). We can define 𝜋 analogously to (4.2). In general, SIMILE requires that Π satisfies a smooth property stated below. This property, which is exploited in our theoretical analysis (see Section 4.5), is motivated by the observation that given a (near) constant stream of context sequence, a stable behavior policy should exhibit a corresponding action sequence with low curvature. The two examples above satisfy this property for sufficiently large 𝜆. 54 Definition 4.2.2 (𝐻-state-smooth imitation policy). For small constant 0 ă 𝐻 ! 1, a policy 𝜋pr𝑥, 𝑎sq is 𝐻-state-smooth if it is 𝐻-smooth w.r.t. 𝑎, i.e. for fixed 𝑥 P X, › › @𝑎, 𝑎 1 P A, @𝑖: ›∇𝜋𝑖 pr𝑥, 𝑎sq ´ ∇𝜋𝑖 pr𝑥, 𝑎 1 sq›˚ ď 𝐻 }𝑎 ´ 𝑎 1 } where 𝜋𝑖 indicates the “ ‰ 𝑖 𝑡ℎ component of vector-valued function3 𝜋p𝑠q “ 𝜋 1 p𝑠q, . . . , 𝜋 𝑘 p𝑠q P R 𝑘 , and }.} and }.}˚ are some norm and dual norm respectively. For twice differentiable policy 𝜋, this is equivalent to having the bound on the Hessian ∇2 𝜋𝑖 pr𝑥, 𝑎sq ĺ 𝐻I 𝑘 @𝑖. 4.3 Related Work in Imitation Learning The most popular traditional approaches for learning from expert demonstration focused on using approximate policy iteration techniques in the MDP setting (Kakade and Langford, 2002; Bagnell et al., 2003). Most prior approaches operate in discrete and finite action space (He et al., 2012; Ratliff et al., 2009; Abbeel and Ng, 2004; Argall et al., 2009). Some focus on continuous state space (Abbeel and Ng, 2005), but requires a linear model for the system dynamics. In contrast, we focus on learning complex smooth functions within continuous action and state spaces. One natural approach to tackle the more general setting is to reduce imitation learning to a standard supervised learning problem (Syed and Schapire, 2010; Langford and Zadrozny, 2005; Lagoudakis and Parr, 2003a). However, standard supervised methods assume i.i.d. training and test examples, thus ignoring the distribution mismatch between training and rolled-out trajectories directly applied to sequential learning problems (Kakade and Langford, 2002). Thus a naive supervised learning approach normally leads to unsatisfactory results (Ross and Bagnell, 2010). Iterative Learning Reductions. State-of-the-art learning reductions for imitation learning typically take an iterative approach, where each training round uses standard supervised learning to learn a policy (Daumé III et al., 2009; Ross et al., 2011a). In each round 𝑛, the following happens: • Given initial state 𝑠0 drawn from the starting distribution of states, the learner executes current policy 𝜋𝑛 , resulting in a sequence of states 𝑠1𝑛 , . . . , 𝑠𝑇𝑛 . • For each 𝑠𝑡𝑛 , a label 𝑎p𝑡𝑛 (e.g., expert feedback) is collected indicating what the expert would do given 𝑠𝑡𝑛 , resulting in a new dataset D𝑛 “ tp𝑠𝑡 , 𝑎p𝑡𝑛 qu. 3 This emphasizes the possibility that 𝜋 is a vector-valued function of 𝑎. The gradient and Hessian are viewed as arrays of 𝑘 gradient vectors and Hessian matrices of 1-d case, since we simply treat action in R 𝑘 as an array of 𝑘 standard functions. 55 • The learner integrates D𝑛 to learn a policy 𝜋ˆ 𝑛 . The learner updates the current policy to 𝜋𝑛`1 based on 𝜋ˆ 𝑛 and 𝜋𝑛 . The main challenge is controlling for the cascading errors caused by the changing dynamics of the system, i.e., the distribution of states in each D𝑛 „ 𝑑 𝜋𝑛 . A policy trained using 𝑑 𝜋𝑛 induces a different distribution of states than 𝑑 𝜋𝑛 , and so is no longer being evaluated on the same distribution as during training. A principled reduction should (approximately) preserve the i.i.d. relationship between training and test examples. Furthermore the state distribution 𝑑 𝜋 should converge to a stationary distribution. The arguably most notable learning reduction approaches for imitation learning are SEARN (Daumé III et al., 2009) and DAgger (Ross et al., 2011a). At each round, SEARN learns a new policy 𝜋ˆ 𝑛 and returns a distribution (or mixture) over previously learned policies: 𝜋𝑛`1 “ 𝛽 𝜋ˆ 𝑛 ` p1 ´ 𝛽q𝜋𝑛 for 𝛽 P p0, 1q. For appropriately small choices of 𝛽, this stochastic mixing limits the “distribution drift” between 𝜋𝑛 and 𝜋𝑛`1 and can provably guarantee that the performance of 𝜋𝑛`1 does not degrage significantly relative to the expert demonstrations.4 DAgger, on the other hand, achieves stability by aggregating a new dataset at each round to learn a new policy from the combined data set D Ð D Y D𝑛 . This aggregation, however, significantly increases the computational complexity and thus is not practical for large problems that require many iterations of learning (since the training time grows super-linearly w.r.t. the number of iterations). Both SEARN and DAgger showed that only a polynomial number of training rounds is required for convergence to a good policy, but with a dependence on the length of horizon 𝑇. In particular, to non-trivially bound the total variation distance }𝑑 𝜋𝑛𝑒𝑤 ´ 𝑑 𝜋𝑜𝑙𝑑 }1 of the state distributions between old and new policies, a learning rate 𝛽 ă 𝑇1 is required to hold (Lemma 1 of Daumé III, Langford, and Marcu (2009) and Theorem 4.1 of Ross, Gordon, and Bagnell (2011a)). As such, systems with very large time horizons might suffer from very slow convergence. Our Contributions. Within the context of previous work, our SIMILE approach can be viewed as extending SEARN to smooth policy classes with the following improvements: 4 A similar approach was adopted in Conservative Policy Iteration for the MDP setting (Kakade and Langford, 2002). 56 • We provide a policy improvement bound that does not depend on the time horizon 𝑇, and can thus converge much faster. In addition, SIMILE has adaptive learning rate, which can further improve convergence. • For the smooth policy class described in Section 4.2, we show how to generate simulated or “virtual” expert feedback in order to guarantee stable learning. This alleviates the need to have continuous access to a dynamic oracle / expert that shows the learner what to do when it is off-track. In this regard, the way SIMILE integrates expert feedback subsumes the set-up from SEARN and DAgger. • Unlike SEARN, SIMILE returns fully deterministic policies. Under the continuous setting, deterministic policies are strictly better than stochastic policies as (i) smoothness is critical and (ii) policy sampling requires holding more data during training, which may not be practical for infinite state and action spaces. • Our theoretical analysis reveals a new sequential prediction setting that yields provably fast convergence, in particular for smooth policy classes on finitehorizon problems. Existing settings that enjoy such results are limited to Markovian dynamics with discounted future rewards or linear model classes. 4.4 Algorithm for Smooth Imitation Learning Setting Our learning algorithm, called SIMILE (Smooth IMItation LEarning), is described in Algorithm 4. At a high level, the process can be described as: 1. Start with some initial policy 𝜋ˆ 0 (Line 2). 2. At iteration 𝑛, use 𝜋𝑛´1 to build a new state distribution S𝑛 and dataset D𝑛 “ tp𝑠𝑡𝑛 , 𝑎p𝑡𝑛 qu (Lines 4-6). 3. Train 𝜋ˆ 𝑛 “ arg min𝜋PΠ E𝑠„S𝑛 rℓ𝑛 p𝜋p𝑠qqs, where ℓ𝑛 is the imitation loss (Lines 7-8). Note that ℓ𝑛 needs not be the original ℓ, but simply needs to converge to it. 4. Interpolate 𝜋ˆ 𝑛 and 𝜋𝑛´1 to generate a new deterministic policy 𝜋𝑛 (Lines 910). Repeat from Step 2 with 𝑛 Ð 𝑛 ` 1 until some termination condition is met. Supervised Learning Reduction. The actual reduction is in Lines 7-8, where we follow a two-step procedure of first updating the smooth regularize ℎ𝑛 , and then 57 Algorithm 4 SIMILE (Smooth IMItation LEarning) Input: features X “ t𝑥𝑡 u, human trajectory A˚ “ t𝑎 𝑡˚ u, base routine Train, smooth regularizers ℎ P H ” ı 1: Initialize A0 Ð A˚ , S0 Ð t 𝑥 𝑡 , 𝑎 𝑡˚´1 u, › 𝑇 › ř › ˚ › ℎ0 “ arg min ›𝑎 𝑡 ´ ℎp𝑎 𝑡˚´1 q› ℎPH 𝑡 “1 2: Initial policy 𝜋0 “ 𝜋ˆ 0 Ð TrainpS0 , A0 | ℎ0 q 3: for 𝑛 “ 1, . . . , 𝑁 do 4: A𝑛 “ t𝑎 𝑡𝑛 u Д 𝜋𝑛´1 pSı𝑛´1 q 5: 6: 7: S𝑛 Ð t𝑠𝑡𝑛 “ 𝑥𝑡 , 𝑎 𝑡𝑛´1 u p 𝑛 “ tp A 𝑎 𝑡𝑛 u @𝑠𝑡𝑛 P S𝑛 › 𝑇 › ř › › 𝑛 𝑎 𝑡𝑛´1 q› ℎ𝑛 “ arg min ›𝑎p𝑡 ´ ℎpp ℎPH 𝑡 “1 p 𝑛 | ℎ𝑛 q 8: 𝜋ˆ 𝑛 Ð TrainpS𝑛 , A 9: 𝛽 Ð 𝛽pℓp𝜋ˆ 𝑛 q, ℓp𝜋𝑛´1 qq 10: 𝜋𝑛 “ 𝛽 𝜋ˆ 𝑛 ` p1 ´ 𝛽q𝜋𝑛´1 11: end for //sequential roll-out ‰ “ 𝑥𝑡:𝑡 ´ 𝑝 , 𝑎 𝑡 ´1:𝑡 ´𝑞 //𝑠𝑡𝑛 “ // collect smooth feedback //new regularizer // train policy //adaptively set 𝛽 // update policy output Last policy 𝜋 𝑁 training 𝜋ˆ 𝑛 via supervised learning. In other words, Train finds the best 𝑓 P F possible for a fixed ℎ𝑛 . We discuss how to set the training targets 𝑎p𝑡𝑛 below. Policy Update. The new policy 𝜋𝑛 is a deterministic interpolation between the previous 𝜋𝑛´1 and the newly learned 𝜋ˆ 𝑛 (Line 10). In contrast, for SEARN, 𝜋𝑛 is a stochastic interploation (Daumé III et al., 2009). Lemma 4.5.2 and Corollary 4.5.3 show that deterministic interpolation converges at least as fast as stochastic for smooth policy classes. This interpolation step plays two key roles. First, it is a form of myopic or greedy online learning. Intuitively, rolling out 𝜋𝑛 leads to incidental exploration on the mistakes of 𝜋𝑛 , and so each round of training is focused on refining 𝜋𝑛 . Second, the interpolation in Line 10 ensures a slow drift in the distribution of states from round to round, which preserves an approximate i.i.d. property for the supervised regression subroutine and guarantees convergence. However this model interpolation creates an inherent tension between maintaining approximate i.i.d. for valid supervised learning and more aggressive exploration (and thus faster convergence). For example, SEARN’s guarantees only apply for small 𝛽 ă 1{𝑇. SIMILE circumvents much of this tension via a policy improvement bound that allows 𝛽 to adaptively increase depending on the quality of 𝜋ˆ 𝑛 (see Theorem 4.5.6), which thus guarantees a valid learning reduction while substantially Anonymous CVPR submission Paper ID 307 060 061 062 063 064 065 066 067 068 069 ract 58 thods are extremely useful in ations. However, the estima070 thin a time independent con071 diction, the results are jittery. ht 072 added after the fact (such as ch is not ideal. Instead, tem-Figure Figure Camera Planning. The objective is to predict (red)073 4.1: 1:Illustration of a smooth simulated feedback vs non-smooth feedback 074 ncorporated into the learning an appropriate pan angle for a broadcast camera based 075 (blue) w how the ‘search and learn’ on noisy player detection data. Consider two planning al076 ed previously for tagging parts gorithms (shown as blue and red curves in the schematic) 077 efficiently learn regressors forspeeding whichup bothconvergence. make the same mistake at time A but recover to a 078 ur data-driven learning techgood framing by C (the ideal camera trajectory is shown in 079 𝑛 oblem: given noisy basketballFeedback black). Generation. The blue solution quickly corrects bytraining time B using We can generate targets 𝑎p𝑡080 using “virtual” feedback arn where the camera should a jerky motion, whereas the red curve conducts a gradual 081 simulating expertthedemonstrations, which has two benefits. First, we need not a human operator. Our exper-from correction. Although red curve has a larger discrepancy 082 arning algorithm which takesquerywithethexpert ˚ ideal motion curve, its velocity characteristics are 𝜋 at every iteration (as done in DAgger 083 (Ross et al., 2011a)). ency of sequential predictions most similar to the ideal motion path. round can be seen as a mance than time independentContinuously acquiring expert demonstrations at every 084 085 special case and a more expensive strategy. Second, virtual086feedback ensures stable data. Often, the estimator is learned within a time indepen087 dent paradigm, and𝜋ˆtemporal regularization is integrated learning, i.e., every smooth policy. as 𝑛 is a feasible 088 a post-processing stage (such as a Kalman filter). However, this two stage approach is notour idealpolicy because𝜋the(blue/red) data-driven made089 Figure 4.1, where a mistake at location A, 𝑛 e the problem of determiningConsider 090 estimator is prevented from learning any temporal patterns. hen broadcasting a basketballand where 091𝜋 ˚ (black). Depending we have only a the single expertestimator demonstration In this paper, we condition data-driven on pre- from camera planning shares many 092 vious predictions, which allows it to learn temporal patterns smoothness requirements of the policy class, we can t tracking: in both cases, theon the 093simulate virtual expert within the data (in addition to any direct feature-based revise an estimated target posi094 lationships). However, formulation (similar feedback as via either thethis redrecursive line (more smooth) or blue (less smooth) as a tradeoff ired. Noise and other ambi095 to reinforcement learning) makes the problem much more trajectories: they are are notbetween squared imitation lossa variant and smoothness. 096 difficult to solve. We employ of the ‘search and objects actually move, and in 097 learn’ (SEARN) algorithm to keep training efficient. Its aesthetic results. In practice,When 098 A˚ , especially during the roll-out of 𝜋 (i.e. A ) differs substantially from 1 recursive𝑛 relationships using an strategy is to decouple𝑛´ the ployed to minimize jitter. The 099 auxiliary reference signal. This allows the predictor to be iterations, using smoother feedback (red instead of 100 blue) can result in more design parameter, and controlsearly learned efficiently using supervised techniques, and our exand smoothness. In contrast tostableperiments 101 learning. We formalize notion forwhen Π𝜆 usin Proposition 4.5.8. Intuitively, demonstrate significantthis improvements s of paramount importance in 102 ing this holistic approach. 𝑠𝑡𝑛 , the feedback should ents which maintain adequatewhenever 𝜋𝑛´1 makes a mistake, resulting in a “bad” state103 tic motions which pursue perrecommend a smooth correction 𝑎p𝑡𝑛 w.r.t. A𝑛 towemake 104 training “easier” for the 105 Problem Definition In the case of camera planning, 𝑛 106expert’s action 𝑎 ˚ . In assumeThe therevirtual is an underlying function f : X 7!converge Y which to the hods, such as random forests,learner.5 feedback 𝑎p𝑡 should 𝑡 107 describes the ideal can be learned directly from 𝑛 camera 𝑛 work that should ˚ occur at the practice, we use 𝑎p𝑡 “ 𝜎𝑎 𝑡 ` p1 ´ 𝜎q𝑎 𝑡 with 𝜎 Ñ 0 as 𝑛 increases (which satisfies Proposition 4.5.8). 1 4.5 Theoretical Analysis of Reduction-Based Algorithm All proofs are deferred to the supplementary material. Stability Conditions One natural smoothness condition is that 𝜋pr𝑥, 𝑎sq should be stable w.r.t. 𝑎 if 𝑥 is fixed. Consider the camera planning setting: the expert policy 𝜋 ˚ should have 5 A similar idea was proposed (He et al., 2012) for DAgger-type algorithm, albeit only for linear model classes. 59 very small curvature, since constant inputs should correspond to constant actions. This motivates Definition 4.2.2, which requires that Π has low curvature given fixed context. We also show that smooth policies per Definition 4.2.2 lead to stable actions, in the sense that “nearby” states are mapped to “nearby” actions. The following helper lemma is useful: Lemma 4.5.1. For a fixed 𝑥, define 𝜋pr𝑥, 𝑎sq fi 𝜑p𝑎q. If 𝜑 is non-negative and 𝐻-smooth w.r.t. 𝑎., then: @𝑎, 𝑎 1 : ` ›2 ˘2 ` ˘› 𝜑p𝑎q ´ 𝜑p𝑎 1 q ď 6𝐻 𝜑p𝑎q ` 𝜑p𝑎 1 q ›𝑎 ´ 𝑎 1 › . “ ‰ Writing 𝜋 as 𝜋pr𝑥, 𝑎sq fi 𝜋 1 pr𝑥, 𝑎sq, . . . , 𝜋 𝑘 pr𝑥, 𝑎sq with each 𝜋𝑖 pr𝑥, 𝑎sq 𝐻-smooth, ? Lemma 4.5.1 implies }p𝜋pr𝑥, 𝑎sq ´ 𝜋pr𝑥, 𝑎 1 sqq} ď 12𝐻𝑅 }𝑎 ´ 𝑎 1 } for 𝑅 upper bounding A. Bounded action space means that a sufficiently small 𝐻 leads to the following stability conditions: Condition 1 (Stability Condition 1). Π satisfies the Stability Condition 1 if for a fixed input feature 𝑥, the actions of 𝜋 in states 𝑠 “ r𝑥, 𝑎s and 𝑠1 “ r𝑥, 𝑎 1 s satisfy }𝜋p𝑠q ´ 𝜋p𝑠1 q} ď }𝑎 ´ 𝑎 1 } for all 𝑎, 𝑎 1 P A. Condition 2 (Stability Condition 2). Π satisfies Stability Condition 2 if each 𝜋 is 𝛾Lipschitz continuous in the action component 𝑎 with 𝛾 ă 1. That is, for a fixed 𝑥 the actions of 𝜋 in states 𝑠 “ r𝑥, 𝑎s and 𝑠1 “ r𝑥, 𝑎 1 s satisfy }𝜋p𝑠q ´ 𝜋p𝑠1 q} ď 𝛾 }𝑎 ´ 𝑎 1 } for all 𝑎, 𝑎 1 P A. These two conditions directly follow from Lemma 4.5.1 and assuming sufficiently small 𝐻. Condition 2 is mildly stronger than Condition 1, and enables proving much stronger policy improvement compared to previous work. Deterministic versus Stochastic Given two policies 𝜋 and 𝜋, ˆ and interpolation parameter 𝛽 P p0, 1q, consider two ways to combine policies: 1. stochastic: 𝜋 𝑠𝑡𝑜 p𝑠q “ 𝜋p𝑠q ˆ with probability 𝛽, and 𝜋 𝑠𝑡𝑜 p𝑠q “ 𝜋p𝑠q with probability 1 ´ 𝛽 2. deterministic: 𝜋 𝑑𝑒𝑡 p𝑠q “ 𝛽 𝜋p𝑠q ˆ ` p1 ´ 𝛽q𝜋p𝑠q Previous learning reduction approaches only use stochastic interpolation (Daumé III et al., 2009; Ross et al., 2011a), whereas SIMILE uses deterministic. The following 60 result shows that deterministic and stochastic interpolation yield the same expected behavior for smooth policy classes. Lemma 4.5.2. Given any starting state 𝑠0 , sequentially execute 𝜋 𝑑𝑒𝑡 and 𝜋 𝑠𝑡𝑜 to obtain two separate trajectories A “ t𝑎 𝑡 u𝑇𝑡“1 and à “ t𝑎˜𝑡 u𝑇𝑡“1 such that 𝑎 𝑡 “ 𝜋 𝑑𝑒𝑡 p𝑠𝑡 q and 𝑎˜ 𝑡 “ 𝜋 𝑠𝑡𝑜 p𝑠˜𝑡 q, where 𝑠𝑡 “ r𝑥𝑡 , 𝑎 𝑡 ´1 s and 𝑠˜𝑡 “ r𝑥𝑡 , 𝑎˜ 𝑡 ´1 s. Assuming the policies are stable as per Condition 1, we have Eà r𝑎˜ 𝑡 s “ 𝑎 𝑡 @𝑡 “ 1, . . . , 𝑇, where the expectation is taken over all random roll-outs of 𝜋 𝑠𝑡𝑜 . Lemma 4.5.2 shows that deterministic policy combination (SIMILE) yields unbiased trajectory roll-outs of stochastic policy combination (as done in SEARN & CPI). This represents a major advantage of SIMILE, since the number of stochastic rollouts of 𝜋 𝑠𝑡𝑜 to average to the deterministic trajectory of 𝜋 𝑑𝑒𝑡 is polynomial in the time horizon 𝑇, leading to much higher computational complexity. Furthermore, for convex imitation loss ℓ𝜋 p𝜋q, Lemma 4.5.2 and Jensen’s inequality yield the following corollary, which states that under convex loss, deterministic policy performs at least no worse than stochastic policy in expectation: Corollary 4.5.3 (Deterministic Policies Perform Better). For deterministic 𝜋 𝑑𝑒𝑡 and stochastic 𝜋 𝑠𝑡𝑜 interpolations of two policies 𝜋 and 𝜋, ˆ and convex loss ℓ, we have: ℓ𝜋 𝑑𝑒𝑡 p𝜋 𝑑𝑒𝑡 q “ ℓ𝜋𝑠𝑡𝑜 pEr𝜋 𝑠𝑡𝑜 sq ď E rℓ𝜋𝑠𝑡𝑜 p𝜋 𝑠𝑡𝑜 qs where the expectation is over all roll-outs of 𝜋 𝑠𝑡𝑜 . Remark. We construct a simple example to show that Condition 1 may be necessary for iterative learning reductions to converge. Consider the case where contexts X Ă R are either constant or vary neglibly. Expert demonstrations should be constant 𝜋 ˚ pr𝑥 𝑛 , 𝑎 ˚ sq “ 𝑎 ˚ for all 𝑛. Consider an unstable policy 𝜋 such that 𝜋p𝑠q “ 𝜋pr𝑥, 𝑎sq “ 𝑘𝑎 for fixed 𝑘 ą 1. The rolled-out trajectory of 𝜋 diverges 𝜋 ˚ at an exponential rate. Assume optimistically that 𝜋ˆ learns the correct expert behavior, which is simply 𝜋p𝑠q ˆ “ 𝜋pr𝑥, ˆ 𝑎sq “ 𝑎. For any 𝛽 P p0, 1q, the updated policy 𝜋 1 “ 𝛽 𝜋ˆ ` p1 ´ 𝛽q𝜋 becomes 𝜋 1 pr𝑥, 𝑎sq “ 𝛽𝑎 ` p1 ´ 𝛽q𝑘𝑎. Thus the sequential roll-out of the new policy 𝜋 1 will also yield an exponential gap from the correct policy. By induction, the same will be true in all future iterations. 61 Policy Improvement Our policy improvement guarantee builds upon the analysis from SEARN (Daumé III et al., 2009), which we extend to using adaptive learning rates 𝛽. We first restate the main policy improvement result from Daumé III et al. (2009). Lemma 4.5.4 (SEARN’s policy nondegradation - Lemma 1 from Daumé III et al. (2009)). Let ℓ𝑚𝑎𝑥 “ sup𝜋,𝑠 ℓp𝜋p𝑠qq, 𝜋 1 is defined as 𝜋 𝑠𝑡𝑜 in lemma 4.5.2. Then for 𝛽 P p0, 1{𝑇q: 1 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝑇E𝑠„𝑑 𝜋 rℓp𝜋p𝑠qqs ˆ ` 𝛽2𝑇 2 ℓ𝑚𝑎𝑥 . 2 SEARN guarantees that the new policy 𝜋 1 does not degrade from the expert 𝜋 ˚ by much only if 𝛽 ă 1{𝑇. Analyses of SEARN and other previous iterative reduction methods (Ross et al., 2011a; Kakade and Langford, 2002; Bagnell et al., 2003; Syed and Schapire, 2010) rely on bounding the variation distance between 𝑑 𝜋 and 𝑑 𝜋1 . Three drawbacks of this approach are: (i) non-trivial variation distance bound typically requires 𝛽 to be inversely proportional to time horizon 𝑇, causing slow convergence; (ii) not easily applicable to the continuous regime; and (iii) except under MDP framework with discounted infinite horizon, previous variation distance bounds do not guarantee monotonic policy improvements (i.e. ℓ𝜋1 p𝜋 1 q ă ℓ𝜋 p𝜋q). We provide two levels of guarantees taking advantage of Stability Conditions 1 and 2 to circumvent these drawbacks. Assuming the Condition 1 and convexity of ℓ, our first result yields a guarantee comparable with SEARN. Theorem 4.5.5 (T-dependent Improvement). Assume ℓ is convex and 𝐿-Lipschitz, and Condition 1 holds. Let 𝜖 “ max }𝜋p𝑠q ˆ ´ 𝜋p𝑠q}. Then: 𝑠„𝑑 𝜋 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝜖 𝐿𝑇 ` 𝛽 pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq . (4.3) In particular, choosing 𝛽 P p0, 1{𝑇q yields: ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝜖 𝐿 ` 𝛽 pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq . (4.4) Similar to SEARN, Theorem 4.5.5 also requires 𝛽 P p0, 1{𝑇q to ensure the RHS of (4.4) stays small. However, note that the reduction term 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq allows the bound to be strictly negative if the policy 𝜋ˆ trained on 𝑑 𝜋 significantly improves on ℓ𝜋 p𝜋q (i.e., guaranteed policy improvement). We observe empirically that this often happens, especially in early iterations of training. 62 Under the mildly stronger Condition 2, we remove the dependency on the time horizon 𝑇, which represents a much stronger guarantee compared to previous work. Theorem 4.5.6 (Policy Improvement). Assume ℓ is convex and 𝐿-Lipschitz continuous, and Condition 2 holds. Let 𝜖 “ max }𝜋p𝑠q ˆ ´ 𝜋p𝑠q}. Then for 𝛽 P p0, 1q: 𝑠„𝑑 𝜋 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝛾𝜖 𝐿 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq. p1 ´ 𝛽qp1 ´ 𝛾q (4.5) Corollary 4.5.7 (Monotonic Improvement). Following the notation from Theorem ´𝛿 , we 4.5.6, let Δ “ ℓ𝜋 p𝜋q ´ ℓ𝜋 p𝜋q ˆ and 𝛿 “ 1𝛾𝜖´𝐿𝛾 . Then choosing step size 𝛽 “ Δ2Δ have: pΔ ´ 𝛿q2 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď ´ . (4.6) 2pΔ ` 𝛿q The terms 𝜖 and ℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋q on the RHS of (4.4) and (4.5) come from the learning reduction, as they measure the “distance” between 𝜋ˆ and 𝜋 on the state distribution induced by 𝜋 (which forms the dataset to train 𝜋). ˆ In practice, both terms can be empirically estimated from the training round, thus allowing an estimate of 𝛽 to minimize the bound. Theorem 4.5.6 justifies using an adaptive and more aggressive interpolation parameter 𝛽 to update policies. In the worst case, setting 𝛽 close to 0 will ensure the bound from (4.5) to be close to 0, which is consistent with SEARN’s result. More generally, monotonic policy improvement can be guaranteed for appropriate choice of 𝛽, as seen from Corollary 4.5.7. This strict policy improvement was not possible under previous iterative learning reduction approaches such as SEARN and DAgger, and is enabled in our setting due to exploiting the smoothness conditions. Smooth Feedback Analysis Smooth Feedback Does Not Hurt: Recall from Section 4.4 that one way to simulate “virtual” feedback for training a new 𝜋ˆ is to set the target 𝑎ˆ 𝑡 “ 𝜎𝑎 𝑡 ` p1 ´ 𝜎q𝑎 𝑡˚ for 𝜎 P p0, 1q, where smooth feedback corresponds to 𝜎 Ñ 1. To see that simulating smooth “virtual” feedback target does not hurt the training progress, we alternatively view SIMILE as performing gradient descent in a smooth function space (Mason et al., 1999). Define the cost functional 𝐶 : Π Ñ R over policy space to be the ş average imitation loss over S as 𝐶p𝜋q “ }𝜋p𝑠q ´ 𝜋 ˚ p𝑠q}2 𝑑𝑃p𝑠q. The gradient S (Gâteaux derivative) of 𝐶p𝜋q w.r.t. 𝜋 is: B𝐶p𝜋 ` 𝛼𝛿 𝑠 q ˇˇ ∇𝐶p𝜋qp𝑠q “ “ 2p𝜋p𝑠q ´ 𝜋 ˚ p𝑠qq, ˇ 𝛼 “0 B𝛼 63 where 𝛿 𝑠 is Dirac delta function centered at s. By first order approximation 𝐶p𝜋 1 q “ 𝐶p𝛽 𝜋ˆ ` p1 ´ 𝛽q𝜋q “ 𝐶p𝜋 ` 𝛽p𝜋ˆ ´ 𝜋qq « 𝐶p𝜋q ` 𝛽x∇𝐶p𝜋q, 𝜋ˆ ´ 𝜋y. Like traditional gradient descent, we want to choose 𝜋ˆ such that the update moves the functional along the direction of negative gradient. In other words, we want to learn 𝜋ˆ P Π such that x∇𝐶p𝜋q, 𝜋ˆ ´ 𝜋y ! 0. We can evaluate this inner product along the states induced by 𝜋. We thus have the estimate: 𝑇 x∇𝐶p𝜋q, 𝜋ˆ ´ 𝜋y « 2ÿ p𝜋p𝑠𝑡 q ´ 𝜋 ˚ p𝑠𝑡 qqp𝜋p𝑠 ˆ 𝑡 q ´ 𝜋p𝑠𝑡 qq 𝑇 𝑡 “1 𝑇 “ 2ÿ p𝑎 𝑡 ´ 𝑎 𝑡˚ qp𝜋pr𝑥 ˆ 𝑡 , 𝑎 𝑡 ´1 sq ´ 𝑎 𝑡 q. 𝑇 𝑡 “1 Since we want x∇𝐶p𝜋q, 𝜋´𝜋y ˆ ă 0, this motivates the construction of new data set D with states tr𝑥𝑡 , 𝑎 𝑡 ´1 su𝑇𝑡“1 and labels tp 𝑎 𝑡 u𝑇𝑡“1 to train a new policy 𝜋, ˆ where we want ˚ p𝑎 𝑡 ´ 𝑎 𝑡 qpp 𝑎 𝑡 ´ 𝑎 𝑡 q ă 0. A sufficient solution is to set target 𝑎p𝑡 “ 𝜎𝑎 𝑡 ` p1 ´ 𝜎q𝑎 𝑡˚ (Section 4.4), as this will point the gradient in negative direction, allowing the learner to make progress. Smooth Feedback is Sometimes Necessary: When the current policy performs poorly, smooth virtual feedback may be required to ensure stable learning, i.e. producing a feasible smooth policy at each training round. We formalize this notion of feasibility by considering the smooth policy class Π𝜆 in Example 4.2.1. Recall that smooth regularization of Π𝜆 via H encourages the next action to be close to the previous action. Thus a natural way to measure smoothness of 𝜋 P Π𝜆 is ř via the average first order difference of consecutive actions 𝑇1 𝑇𝑡“1 }𝑎 𝑡 ´ 𝑎 𝑡 ´1 }. In particular, we want to explicitly constrain this difference relative ›to the expert › › 1 ř𝑇 1 ř𝑇 › ˚ ˚ trajectory 𝑇 𝑡 “1 }𝑎 𝑡 ´ 𝑎 𝑡 ´1 } ď 𝜂 at each iteration, where 𝜂9 𝑇 𝑡 “1 ›𝑎 𝑡 ´ 𝑎 𝑡 ´1 ›. When 𝜋 performs poorly, i.e. the "average gap" between current trajectory t𝑎 𝑡 u and t𝑎 𝑡˚ u is large, the training target for 𝜋ˆ should be lowered to ensure learning a smooth policy is feasible, as inferred from the following proposition. In practice, we typically employ smooth virtual feedback in early iterations when policies tend to perform worse. Proposition 4.5.8. Let 𝜔 be the average supervised training error from F , i.e. 𝜔 “ min E𝑥 „X r} 𝑓 pr𝑥, 0sq ´ 𝑎 ˚ }s. Let the rolled-out trajectory of current policy 𝜋 be 𝑓 PF t𝑎 𝑡 u. If the average gap between 𝜋 and 𝜋 ˚ is such that E𝑡 „Uniformr1:𝑇 s r}𝑎 𝑡˚ ´ 𝑎 𝑡 ´1 }s ě 3𝜔 ` 𝜂p1 ` 𝜆q, then using t𝑎 𝑡˚ u as feedback will cause the trained policy 𝜋ˆ to be 64 non-smooth, i.e.: E𝑡 „Uniformr1:𝑇 s r}𝑎ˆ 𝑡 ´ 𝑎ˆ 𝑡 ´1 }s ě 𝜂, (4.7) for t𝑎ˆ 𝑡 u the rolled-out trajectory of 𝜋. ˆ 4.6 Experiments Automated Camera Planning. We evaluate SIMILE in a case study of automated camera planning for sport broadcasting (Chen and Carr, 2015; Chen et al., 2016). Given noisy tracking of players as raw input data t𝑥𝑡 u𝑇𝑡“1 , and demonstrated pan camera angles from professional human operator as t𝑎 𝑡˚ u𝑇𝑡“1 , the goal is to learn a policy 𝜋 that produces trajectory t𝑎 𝑡 u𝑇𝑡“1 that is both smooth and accurate relative to t𝑎 𝑡˚ u𝑇𝑡“1 . Smoothness is critical in camera control: fluid movements which maintain adequate framing are preferable to jittery motions which constantly pursue perfect tracking (Gaddam et al., 2015). In this setting, time horizon 𝑇 is the duration of the event multiplied by rate of sampling. Thus 𝑇 tends to be very large. Smooth Policy Class. We use a smooth policy class following Example 4.2.2: regression tree ensembles F regularized by a class of linear autoregressor functions H (Chen et al., 2016). See Appendix C.2 for more details. Summary of Results. • Using our smooth policy class leads to dramatically smoother trajectories than not regularizing using H . • Using our adaptive learning rate leads to much faster convergence compared to conservative learning rates from SEARN (Daumé III et al., 2009). • Using smooth feedback ensures stable learning of smooth policies at each iteration. • Deterministic policy interpolation performs better than stochastic interpolation used in SEARN. Smooth versus Non-Smooth Policy Classes. Figure 4.2 shows a comparison of using a smooth policy class versus a non-smooth one (e.g., not using H ). We see that our approach can reliably learn to predict trajectories that are both smooth and accurate. Adaptive vs. Fixed 𝛽: One can, in principle, train using SEARN, which requires a very conservative 𝛽 in order to guarantee convergence. In contrast, SIMILE adaptively selects 𝛽 based on relative empirical loss of 𝜋 and 𝜋ˆ (Line 9 of Algorithm 4). Let errorp𝜋q ˆ and errorp𝜋q denote the mean-squared errors of rolled-out 65 Figure 4.2: Expert (blue) and predicted (red) camera pan angles. Left: SIMILE with ă10 iterations. Right: non-smooth policy. Figure 4.3: Adaptive versus fixed interpolation parameter 𝛽. trajectories t𝑎ˆ 𝑡 u, t𝑎 𝑡 u, respectively, w.r.t. ground truth t𝑎 𝑡˚ u. We can set 𝛽 as: 𝛽ˆ “ errorp𝜋q , errorp𝜋q ˆ ` errorp𝜋q (4.8) which encourages the learner to disregard bad policies when interpolating, thus allowing fast convergence to a good policy (see Theorem 4.5.6). Figure 4.3 compares the convergence rate of SIMILE using adaptive 𝛽 versus conservative fixed values of 𝛽 commonly used in SEARN (Daumé III et al., 2009). We see that adaptively choosing 𝛽 enjoys substantially faster convergence. Note that very large fixed 𝛽 may overshoot and worsen the combined policy after a few initial improvements. Smooth Feedback Generation: We set the target labels to 𝑎ˆ 𝑡𝑛 “ 𝜎𝑎 𝑡𝑛 ` p1 ´ 𝜎q𝑎 𝑡˚ for 0 ă 𝜎 ă 1 (Line 6 of Algorithm 4). Larger 𝜎 corresponds to smoother (𝑎ˆ 𝑡𝑛 66 Figure 4.4: Comparing different values of 𝜎. ht Figure 4.5: Smoothness Structure vs Accuracy Trade-off is closer to 𝑎 𝑡𝑛´1 ) but less accurate target (further from 𝑎 𝑡˚ ), as seen in Figure 4.4. Figure 4.5 shows the trade-off between smoothness loss (blue line, measured by first order difference in Proposition 4.5.8) and imitation loss (red line, measured by mean squared distance) for varying 𝜎. We navigate this trade-off by setting 𝜎 closer to 1 in early iterations, and have 𝜎 Ñ 0 as 𝑛 increases. This “gradual increase” produces more stable policies, especially during early iterations where the learning policy tends to perform poorly (as formalized in Proposition 4.5.8). In Figure 4.4, when the initial policy (green trajectory) has poor performance, setting smooth targets (Figure 4.4b) allows learning a smooth policy in the subsequent round, in contrast to more accurate but less stable performance of “difficult” targets with low 𝜎 (Figure 67 4.4c-d). Figure 4.6 visualizes the behavior of the the intermediate policies learned by SIMILE, where we can see that each intermediate policy is a smooth policy. Figure 4.6: Performance after different number of iterations. Deterministic vs. Stochastic Interpolation: Finally, we evaluate the benefits of using deterministic policy averaging intead of stochastically combine different policies, as done in SEARN. To control for other factors, we set 𝛽 to a fixed value of 0.5, and keep the new training dataset D𝑛 the same for each iteration 𝑛. The average imitation loss of stochastic policy sampling are evaluated after 50 stochastic roll-outs at each iterations. This average stochastic policy error tends to be higher compared to the empirical error of the deterministic trajectory, as seen from Figure 4.7, and confirms our finding from Corollary 4.5.3. Figure 4.7: Deterministic policy error vs. average stochastic policy error for 𝛽 “ 0.5 and 50 roll-outs of the stochastic policies. 68 4.7 Discussion We formalized the problem of smooth imitation learning for online sequence prediction, which is a variant of imitation learning that uses a notion of a smooth policy class. We proposed SIMILE (Smooth IMItation LEarning), which is an iterative learning reduction approach to learning smooth policies from expert demonstrations in a continuous and dynamic environment. SIMILE utilizes an adaptive learning rate that provably allows much faster convergence compared to previous learning reduction approaches, and also enjoys better sample complexity than previous work by being fully deterministic and allowing for virtual simulation of training labels. We validated the efficiency and practicality of our approach on a setting of online camera planning. 69 Chapter 5 REDUCTION TO ONLINE LEARNING WITH POLICY-BASED STRUCTURE (PROGRAMMATIC REINFORCEMENT LEARNING) Summary. We study the problem of programmatic reinforcement learning, in which policies are represented as short programs in a symbolic language. Programmatic policies can be more interpretable, generalizable, and amenable to formal verification than neural policies; however, designing rigorous learning approaches for such policies remains a challenge. Our approach to this challenge — a meta-algorithm called Propel— is based on three insights. First, we view our learning task as optimization in policy space, modulo the constraint that the desired policy has a programmatic representation, and solve this optimization problem using a form of mirror descent that takes a gradient step into the unconstrained policy space and then projects back onto the constrained space. Second, we view the unconstrained policy space as mixing neural and programmatic representations, which enables employing state-of-the-art deep policy gradient approaches. Third, we cast the projection step as program synthesis via imitation learning, and exploit contemporary combinatorial methods for this task. We present theoretical convergence results for Propel and empirically evaluate the approach in three continuous control domains. The experiments show that Propel can significantly outperform state-of-the-art approaches for learning programmatic policies. 5.1 Introduction to Programmatic Reinforcement Learning A growing body of work (Verma et al., 2018; Bastani et al., 2018; Zhu et al., 2019) investigates reinforcement learning (RL) approaches that represent policies as programs in a symbolic language, e.g., a domain-specific language for composing control modules such as PID controllers (Ang et al., 2005). Short programmatic policies offer many advantages over neural policies discovered through deep RL, including greater interpretability, better generalization to unseen environments, and greater amenability to formal verification. These benefits motivate developing effective approaches for learning such programmatic policies. However, programmatic reinforcement learning (PRl) remains a challenging problem, owing to the highly structured nature of the policy space. Recent state-of-the-art 70 approaches employ program synthesis methods to imitate or distill a pre-trained neural policy into short programs (Verma et al., 2018; Bastani et al., 2018). However, such a distillation process can yield a highly suboptimal programmatic policy — i.e., a large distillation gap — and the issue of direct policy search for programmatic policies also remains open. In this chapter, we develop Propel (Imitation-Projected Programmatic Reinforcement Learning), a new learning meta-algorithm for PRl, as a response to this challenge. The design of Propel is based on three insights that enables integrating and building upon state-of-the-art approaches for policy gradients and program synthesis. First, we view programmatic policy learning as a constrained policy optimization problem, in which the desired policies are constrained to be those that have a programmatic representation. This insight motivates utilizing constrained mirror descent approaches, which take a gradient step into the unconstrained policy space and then project back onto the constrained space. Second, by allowing the unconstrained policy space to have a mix of neural and programmatic representations, we can employ well-developed deep policy gradient approaches (Sutton et al., 2000; Lillicrap et al., 2015; Schulman et al., 2015, 2017; Cheng et al., 2019e) to compute the unconstrained gradient step. Third, we define the projection operator using program synthesis via imitation learning (Verma et al., 2018; Bastani et al., 2018), in order to recover a programmatic policy from the unconstrained policy space. Our contributions can be summarized as: • We present Propel, a novel meta-algorithm that is based on mirror descent, program synthesis, and imitation learning, for PRl. • On the theoretical side, we show how to cast Propel as a form of constrained mirror descent. We provide a thorough theoretical analysis characterizing the impact of approximate gradients and projections. Further, we prove results that provide expected regret bounds and finite-sample guarantees under reasonable assumptions. • On the practical side, we provide a concrete instantiation of Propel and evaluate it in three continuous control domains, including the challenging car-racing domain Torcs (Wymann et al., 2014). The experiments show significant improvements over state-of-the-art approaches for learning programmatic policies. 71 5.2 Policy Learning Problem within the Structured Policy Class The problem of programmatic reinforcement learning (PRl) consists of a Markov Decision Process (Mdp) M and a programmatic policy class Π. The definition of M “ pS, A, 𝑃, 𝑐, 𝑝 0 , 𝛾q is standard (Sutton and Barto, 2018b), with S being the state space, A the action space, 𝑃p𝑠1 |𝑠, 𝑎q the probability density function of transitioning from a state-action pair to a new state, 𝑐p𝑠, 𝑎q the state-action cost function, 𝑝 0 p𝑠q a distribution over starting states, and 𝛾 P p0, 1q the discount factor. A policy 𝜋 : S Ñ A (stochastically) maps states to actions. We focus on continuous control problems, so S and A are assumed to be continuous spaces. The goal is to find a programmatic policy 𝜋 ˚ P Π such that: « ff 8 ÿ (5.1) where: 𝐽p𝜋q “ E 𝛾 𝑖 𝑐p𝑠𝑖 , 𝑎𝑖 ” 𝜋p𝑠𝑖 qq , 𝜋 ˚ “ arg min 𝐽p𝜋q, 𝜋 PΠ 𝑖 “0 with the expectation taken over the initial state distribution 𝑠0 „ 𝑝 0 , the policy decisions, and the transition dynamics 𝑃. One can also use rewards, in which case (5.1) becomes a maximization problem. Programmatic Policy Class. A programmatic policy class Π consists of policies that can be represented parsimoniously by a (domain-specific) programming language. Recent work (Verma et al., 2018; Bastani et al., 2018; Zhu et al., 2019) indicates that such policies can be easier to interpret and formally verify than neural policies, and can also be more robust to changes in the environment. In this chapter, we consider two concrete classes of programmatic policies. The first, a simplification of the class considered in Verma et al. (Verma et al., 2018), is defined by the modular, high-level language in Figure 5.1. This language assumes a library of parameterized functions ‘𝜃 representing standard controllers, for instance Proportional-Integral-Derivative (PID) (Åström and Hägglund, 1984) or bang-bang controllers (Bellman et al., 1956). Programs in the language take states 𝑠 as inputs and produce actions 𝑎 as output, and can invoke fully instantiated library controllers along with predefined arithmetic, boolean and relational operators. The second, “lower-level" class, from Bastani et al. (Bastani et al., 2018), consists of decision trees that map states to actions. Example. Consider the problem of learning a programmatic policy, in the language of Figure 5.1, that controls a car’s accelerator in the Torcs car-racing environment (Wymann et al., 2014). Figure 5.2 shows a program in our language for this task. The program invokes PID controllers PIDx 𝑗,𝜃 𝑃 ,𝜃 𝐼 ,𝜃 𝐷 y , where 𝑗 identifies the 72 𝜋p𝑠q ::“ 𝑎 | Opp𝜋1 p𝑠q, . . . , 𝜋 𝑘 p𝑠qq | if 𝑏 then 𝜋1 p𝑠q else 𝜋2 p𝑠q | ‘ 𝜃 p𝜋1 p𝑠q, . . . , 𝜋 𝑘 p𝑠qq 𝑏 ::“ 𝜙p𝑠q | BOpp𝑏 1 , . . . , 𝑏 𝑘 q Figure 5.1: A high-level syntax for programmatic policies, inspired by (Verma et al., 2018). A policy 𝜋p𝑠q takes a state 𝑠 as input and produces an action 𝑎 as output. 𝑏 represents boolean expressions; 𝜙 is a boolean-valued operator on states; Op is an operator that combines multiple policies into one policy; BOp is a standard boolean operator; and ‘𝜃 is a “library function" parameterized by 𝜃. if p𝑠rTrackPoss ă 0.011 and 𝑠rTrackPoss ą ´0.011q then PIDxRPM,0.45,3.54,0.03,53.39y p𝑠q else PIDxRPM,0.39,3.54,0.03,53.39y p𝑠q Figure 5.2: A programmatic policy for acceleration in Torcs (Wymann et al., 2014), automatically discovered by Propel. 𝑠rTrackPoss represents the most recent reading from sensor TrackPos. sensor (out of 29, in our experiments) that provides inputs to the controller, and 𝜃 𝑃 , 𝜃 𝐼 , and 𝜃 𝐷 are respectively the real-valued coefficients of the proportional, integral, and derivative terms in the controller. We note that the program only uses the sensors TrackPos and RPM. While TrackPos (for the position of the car relative to the track axis) is used to decide which controller to use, only the RPM sensor is needed to calculate the acceleration. Learning Challenges. Learning programmatic policies in the continuous RL setting is challenging, as the best performing methods utilize policy gradient approaches (Sutton et al., 2000; Lillicrap et al., 2015; Schulman et al., 2015, 2017; Cheng et al., 2019e), but policy gradients are hard to compute in programmatic representations. In many cases, Π may not even be differentiable. For our approach, we only assume access to program synthesis methods that can select a programmatic policy 𝜋 P Π that minimizes imitation disagreement with demonstrations provided by a teaching oracle. Because imitation learning tends to be easier than general RL in longhorizon tasks (Sun et al., 2017), the task of imitating a neural policy with a program is, intuitively, significantly simpler than the full programmatic RL problem. This intuition is corroborated by past work on programmatic RL (Verma et al., 2018), which shows that direct search over programs often fails to meet basic performance objectives. 73 Algorithm 5 Imitation-Projected Programmatic Reinforcement Learning (Propel) 1: Input: Programmatic & Neural Policy Classes: Π & F . 2: Input: Either initial 𝜋0 or initial 𝑓0 3: Define joint policy class: H ” Π ‘ F //ℎ ” 𝜋 ` 𝑓 defined as ℎp𝑠q “ 𝜋p𝑠q ` 𝑓 p𝑠q 4: if given initial 𝑓0 then 5: 𝜋0 Ð Projectp 𝑓0 q //program synthesis via imitation learning 6: end if 7: for 𝑡 “ 1, . . . , 𝑇 do 8: ℎ𝑡 Ð Update F p𝜋𝑡´1 , 𝜂q //policy gradient in neural policy space with learning rate 𝜂 9: 𝜋𝑡 Ð ProjectΠ pℎ𝑡 q 10: end for 11: Return: Policy 𝜋𝑇 5.3 //program synthesis via imitation learning Learning Algorithm via Reduction to Online Learning To develop our approach, we take the viewpoint of (5.1) being a constrained optimization problem, where Π Ă H resides within a larger space of policies H . In particular, we will represent H ” Π ‘ F using a mixing of programmatic policies Π and neural polices F . Any mixed policy ℎ ” 𝜋 ` 𝑓 can be invoked as ℎp𝑠q “ 𝜋p𝑠q ` 𝑓 p𝑠q. In general, we assume that F is a good approximation of Π (i.e., for each 𝜋 P Π there is some 𝑓 P F that approximates it well), which we formalize in Section 5.4. We can now frame our constrained learning problem as minimizing (5.1) over Π Ă H , that alternate between taking a gradient step in the general space H and projecting back down onto Π. This “lift-and-project” perspective motivates viewing our problem via the lens of mirror descent (Nemirovsky and Yudin, 1983). In standard mirror descent, the unconstrained gradient step can be written as ℎ Ð ℎ 𝑝𝑟𝑒𝑣 ´ 𝜂∇H 𝐽pℎ 𝑝𝑟𝑒𝑣 q for step size 𝜂, and the projection can be written as 𝜋 Ð arg min𝜋1 PΠ 𝐷p𝜋 1 , ℎq for divergence measure 𝐷. Our approach, Imitation-Projected Programmatic Reinforcement Learning (Propel), is outlined in Algorithm 5 (also see Figure 5.3). Propel is a meta-algorithm that requires instantiating two subroutines, Update and Project, which correspond to the standard update and projection steps, respectively. Propel can be viewed as a form of functional mirror descent with some notable deviations from vanilla mirror descent. UpdateF . Since policy gradient methods are well-developed for neural policy classes F (e.g., (Lillicrap et al., 2015; Schulman et al., 2015, 2017; Henderson et al., 2018; Duan et al., 2016; Cheng et al., 2019e)) and non-existent for programmatic H 74 <latexit sha1_base64="FjVpgZ9sYnF4JX/Xxf2CItINeeg=">AAAB8nicbVDLSgMxFM3UV62vqks3wSK4KjNV0GXRTZcV7AOmQ8mkmTY0kwzJHaEM/Qw3LhRx69e482/MtLPQ1gOBwzn3knNPmAhuwHW/ndLG5tb2Tnm3srd/cHhUPT7pGpVqyjpUCaX7ITFMcMk6wEGwfqIZiUPBeuH0Pvd7T0wbruQjzBIWxGQsecQpASv5g5jAhBKRtebDas2tuwvgdeIVpIYKtIfVr8FI0TRmEqggxviem0CQEQ2cCjavDFLDEkKnZMx8SyWJmQmyReQ5vrDKCEdK2ycBL9TfGxmJjZnFoZ3MI5pVLxf/8/wUotsg4zJJgUm6/ChKBQaF8/vxiGtGQcwsIVRzmxXTCdGEgm2pYkvwVk9eJ91G3buqNx6ua827oo4yOkPn6BJ56AY1UQu1UQdRpNAzekVvDjgvzrvzsRwtOcXOKfoD5/MHfD+RYg==</latexit> Update FUpdate F Update F Update F <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> ⇧ Update F <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> <latexit sha1_base64="wssQ+renNbqLlT7M0IUVY1jBbiA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeClx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNM7uZ+54lrI2L1iNOE+xEdKREKRtFKD/2mGJQrbtVdgKwTLycVyNEclL/6w5ilEVfIJDWm57kJ+hnVKJjks1I/NTyhbEJHvGepohE3frY4dUYurDIkYaxtKSQL9fdERiNjplFgOyOKY7PqzcX/vF6K4a2fCZWkyBVbLgpTSTAm87/JUGjOUE4toUwLeythY6opQ5tOyYbgrb68Ttq1qndVrd1fV+qNPI4inME5XIIHN1CHBjShBQxG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAiOY23</latexit> Update F <latexit sha1_base64="uVEU6tZiG0qwOqgTWDNX6SrYYHs=">AAACBnicbVBNS8NAEN34WetX1KMIwSJ4KkkV7LEgiMcKpi20IWw2m3bp5oPdiVBCTl78K148KOLV3+DNf+MmzUFbHww83pthZp6XcCbBNL+1ldW19Y3N2lZ9e2d3b18/OOzJOBWE2iTmsRh4WFLOImoDA04HiaA49Djte9Prwu8/UCFZHN3DLKFOiMcRCxjBoCRXPxmFGCYMMjvxMdDczUqBYJ7d5LmrN8ymWcJYJlZFGqhC19W/Rn5M0pBGQDiWcmiZCTgZFsAIp3l9lEqaYDLFYzpUNMIhlU5WvpEbZ0rxjSAWqiIwSvX3RIZDKWehpzqLG+WiV4j/ecMUgraTsShJgUZkvihIuQGxUWRi+ExQAnymCCaCqVsNMsECE1DJ1VUI1uLLy6TXaloXzdbdZaPTruKooWN0is6Rha5QB92iLrIRQY/oGb2iN+1Je9HetY9564pWzRyhP9A+fwCaZZnQ</latexit> Project ⇧ <latexit sha1_base64="yJrGS/uwBUglQp5t/VvEbT6FBk0=">AAACD3icbVC7SgNBFJ2Nrxhfq5Y2i0GxCrtR0DJoYxnBPCAbwuzkJhkz+2DmrhiW/QMbf8XGQhFbWzv/xtlkC008MHA49zHnHi8SXKFtfxuFpeWV1bXiemljc2t7x9zda6owlgwaLBShbHtUgeABNJCjgHYkgfqegJY3vsrqrXuQiofBLU4i6Pp0GPABZxS11DOPXYQHnO5JPBFDmrg+xRHHpC7DO2CY9tw6T3tm2a7YU1iLxMlJmeSo98wvtx+y2IcAmaBKdRw7wm5CJXImIC25sYKIsjEdQkfTgPqgusnUR2odaaVvDUKpX4DWVP09kVBfqYnv6c7MrJqvZeJ/tU6Mg4tuwoMoRgjY7KNBLCwMrSwcq8+lPllMNKFMcu3VYiMqKUMdYUmH4MyfvEia1YpzWqnenJVrl3kcRXJADskJccg5qZFrUicNwsgjeSav5M14Ml6Md+Nj1low8pl98gfG5w9vop4p</latexit> Figure 5.3: Depicting the Propel meta-algorithm. policy classes Π, Propel is designed to leverage policy gradients in F and avoid policy gradients in Π. Algorithm 6 shows one instantiation of UpdateF . Note that standard mirror descent takes unconstrained gradient steps in H rather than F , and we discuss this discrepancy between UpdateH and UpdateF in Section 5.4. ProjectΠ . Projecting onto Π can be implemented using program synthesis via imitation learning, i.e., by synthesizing a 𝜋 P Π to best imitate demonstrations provided by a teaching oracle ℎ P H . Recent work (Verma et al., 2018; Bastani et al., 2018; Zhu et al., 2019) has given practical heuristics for this task for various programmatic policy classes. Algorithm 7 shows one instantiation of ProjectΠ (based on DAgger (Ross et al., 2011a)). One complication that arises is that finitesample runs of such imitation learning approaches only return approximate solutions and so the projection is not exact. We characterize the impact of approximate projections in Section 5.4. Practical Considerations. In practice, we often employ multiple gradient steps before taking a projection step (as also described in Algorithm 6), because the step size of individual (stochastic) gradient updates can be quite small. Another issue that arises in virtually all policy gradient approaches is that the gradient estimates can have very high variance (Sutton et al., 2000; Konda and Tsitsiklis, 2000; Henderson et al., 2018). We utilize low-variance policy gradient updates by using the reference 𝜋 as a proximal regularizer in function space (Cheng et al., 2019e). For the projection step (Algorithm 7), in practice we often retain all previous roll-outs 𝜏 from all previous projection steps. It is straightforward to query the current oracle ℎ to provide demonstrations on the states 𝑠 P 𝜏 from previous roll-outs, which can lead to substantial savings in sample complexity with regards to executing roll-outs on the environment, while not harming convergence. 75 Algorithm 6 UpdateF : neural policy gradient for mixed policies 1: Input: Neural Policy Class F . Input: Reference programmatic policy: 𝜋 2: Input: Step size: 𝜂. Input: Regularization parameter: 𝜆 3: Initialize neural policy: 𝑓0 //any standard randomized initialization 4: for 𝑗 “ 1, . . . , 𝑚 do 5: 𝑓 𝑗 Ð 𝑓 𝑗´1 ´ 𝜂𝜆∇ F 𝐽p𝜋 ` 𝜆 𝑓 𝑗´1 q //using DDPG (Lillicrap et al., 2015), TRPO (Schulman et al., 2015), etc., holding 𝜋 fixed 6: end for 7: Return: ℎ ” 𝜋 ` 𝜆 𝑓𝑚 Algorithm 7 ProjectΠ : program synthesis via imitation learning 1: Input: Programmatic Policy Class: Π. Input: Oracle policy: ℎ 2: Roll-out ℎ on environment, get trajectory: 𝜏0 “ p𝑠0 , ℎp𝑠0 q, 𝑠1 , ℎp𝑠1 q, . . .q 3: Create supervised demonstration set: Γ0 “ tp𝑠, ℎp𝑠qqu from 𝜏0 4: Derive 𝜋0 from Γ0 via program synthesis //e.g., using methods in (Verma et al., 2018; Bastani et al., 2018) 5: for 𝑘 “ 1, . . . , 𝑀 do 6: Roll-out 𝜋 𝑘´1 , creating trajectory: 𝜏𝑘 7: Collect demonstration data: Γ1 “ tp𝑠, ℎp𝑠qq|𝑠 P 𝜏𝑘 u 8: Γ 𝑘 Ð Γ1 Y Γ 𝑘´1 //DAgger-style imitation learning (Ross et al., 2011a) 9: Derive 𝜋 𝑘 from Γ 𝑘 via program synthesis //e.g., using methods in (Verma et al., 2018; Bastani et al., 2018) 10: end for 11: Return: 𝜋 𝑀 5.4 Theoretical Analysis We start by viewing Propel through the lens of online learning in function space, independent of the specific parametric representation. This start point yields a convergence analysis of Alg. 5 in Section 5.4 under generic approximation errors. We then analyze the issues of policy class representation in Sections 5.4 and 5.4, and connect Algorithms 6 and 7 with the overall performance, under some simplifying conditions. In particular, Section 5.4 characterizes the update error in a possibly non-differentiable setting; to our knowledge, this is the first such analysis of its kind for reinforcement learning. Preliminaries. We consider Π and F to be subspaces of an ambient policy space U, which is a vector space equipped with inner product x¨, ¨y, induced norm }𝑢} “ a x𝑢, 𝑢y, dual norm }𝑣}˚ “ suptx𝑣, 𝑢y| }𝑢} ď 1u, and standard scaling & addition: p𝑎𝑢 ` 𝑏𝑣qp𝑠q “ 𝑎𝑢p𝑠q ` 𝑏𝑣p𝑠q for 𝑎, 𝑏 P R and 𝑢, 𝑣 P U. The cost functional of a ş policy 𝑢 is 𝐽p𝑢q “ S 𝑐p𝑠, 𝑢p𝑠qq𝑑𝜇𝑢 p𝑠q, where 𝜇𝑢 is the distribution of states induced by 𝑢. The joint policy class is H “ Π ‘ F , by H “ t𝜋 ` 𝑓 |@𝜋 P Π, 𝑓 P F u.1 Note that H is a subspace of U, and inherits its vector space properties. Without 1 The operator ‘ is not a direct sum, since Π and F are not orthogonal. 76 affecting the analysis, we simply equate U ” H for the remainder of the chapter. We assume that 𝐽 is convex in H , which implies that subgradient B𝐽pℎq exists (with respect to H ) (Bauschke et al., 2011). Where 𝐽 is differentiable, we utilize the notion of a Fréchet gradient. Recall that a bounded linear operator ∇ : H ÞÑ H is 𝐽 p ℎ`𝑔 q´𝐽 p ℎq´x∇𝐽 p ℎq,𝑔 y called a Fréchet functional gradient of 𝐽 at ℎ P H if lim “ 0. }𝑔 } }𝑔}Ñ0 By default, ∇ (or ∇H for emphasis) denotes the gradient with respect to H , whereas ∇F defines the gradient in the restricted subspace F . Propel as (Approximate) Functional Mirror Descent For our analysis, Propel can be viewed as approximating mirror descent in (infinitedimensional) function space over a convex set Π Ă H .2 Similar to the finitedimensional setting (Nemirovsky and Yudin, 1983), we choose a strongly convex and smooth functional regularizer 𝑅 to be the mirror map. From the approximate mirror descent perspective, for each iteration 𝑡: p 𝑡 ´1 « ∇𝐽p𝜋𝑡 ´1 q 1. Obtain a noisy gradient estimate: ∇ p 𝑡 ´1 2. UpdateH p𝜋q in H space: ∇𝑅pℎ𝑡 q “ ∇𝑅p𝜋𝑡 ´1 q ´ 𝜂 ∇ UpdateH ‰ UpdateF ) (Note 3. Obtain approximate projection: 𝜋𝑡 “ ProjectΠ𝑅 pℎ𝑡 q « arg min𝜋PΠ 𝐷 𝑅 p𝜋, ℎ𝑡 q 𝐷 𝑅 p𝑢, 𝑣q “ 𝑅p𝑢q ´ 𝑅p𝑣q ´ x∇𝑅p𝑢q, 𝑢 ´ 𝑣y is a Bregman divergence. Taking 𝑅pℎq “ 2 1 2 }ℎ} will recover projected functional gradient descent in 𝐿2-space. Here Update 2 p becomes ℎ𝑡 “ 𝜋𝑡 ´1 ´ 𝜂 ∇𝐽p𝜋 𝑡 ´1 q, and Project solves for arg min𝜋 PΠ }𝜋 ´ ℎ𝑡 } . While we mainly focus on this choice of 𝑅 in our experiments, note that other selections of 𝑅 lead to different Update and Project operators (e.g., minimizing KL divergence if 𝑅 is negative entropy). The functional mirror descent scheme above may encounter two additional sources of error compared to standard mirror descent (Nemirovsky and Yudin, 1983). First, in the stochastic setting (also called bandit feedback (Flaxman et al., 2005)), the p 𝑡 may be biased, in addition to having high variance. One gradient estimate ∇ potential source of bias is the gap between UpdateH and UpdateF . Second, the Project step may be inexact. We start by analyzing the behavior of Propel under generic bias, variance, and projection errors, before discussing the implications of approximating UpdateH and ProjectΠ by Algs. 6 & 7, respectively. Let the bias 2 Π can be convexified by considering randomized policies, as stochastic combinations of 𝜋 P Π (cf. (Le et al., 2019b)). 77 › › › p › be bounded by 𝛽, i.e., ›Er∇ 𝑡 |𝜋𝑡 s ´ ∇𝐽p𝜋𝑡 q› ď 𝛽 almost surely. Similarly let the ˚ variance of the gradient estimate be bounded by 𝜎 2 , and the projection error norm }𝜋𝑡 ´ 𝜋𝑡˚ } ď 𝜖. We state the expected regret bound below; more details and a proof appear in Appendix D.1. Theorem 5.4.1 (Expected regret bound under gradient estimation and projection errors). Let 𝜋1 , . . . , 𝜋𝑇 be a sequence of programmatic policies returned by Al˚ gorithm b 5, and 𝜋 be the optimal programmatic policy. Choosing learning rate 𝜂 “ 𝜎12 p 𝑇1 ` 𝜖q, we have the expected regret over 𝑇 iterations: ¸ « ff ˜ c 𝑇 1 1ÿ `𝜖 `𝛽 . (5.2) E 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝜎 𝑇 𝑡 “1 𝑇 The result shows that error 𝜖 from Project and the bias 𝛽 do not accumulate and simply contribute an additive term on the expected regret.3 The effect of variance of a gradient estimate decreases at a 1{𝑇 rate. Note that this regret bound is agnostic to the specific Update and Project operations, and can be applied more generically beyond the specific algorithmic choices used in this section of the thesis. Finite-Sample Analysis under Vanilla Policy Gradient Update and DAgger Projection Next, we show how certain instantiations of Update and Project affect the magnitude of errors and influence end-to-end learning performance from finite samples, under some simplifying assumptions on the Update step. For this analysis, we simplify Alg. 6 into the case UpdateF ” UpdateH . In particular, we assume programmatic policies in Π to be parameterized by a vector 𝜃 P R 𝑘 , and 𝜋 is differentiable in 𝜃 (e.g., we can view Π Ă F where F is parameterized in R 𝑘 ). We further assume the trajectory roll-out is performed in an exploratory manner, where action is taken uniformly random over finite set of 𝐴 actions, thus enabling the bound on the bias of gradient estimates via Bernstein’s inequality. The Project step is consistent with Alg. 7, i.e., using DAgger (Ross et al., 2011b) under convex imitation loss, such as ℓ2 loss. We have the following high-probability guarantee: Theorem 5.4.2 (Finite-sample guarantee). At each iteration, we perform vanilla policy gradient estimate of 𝜋 (over H ) using 𝑚 trajectories and, use DAgger algorithm to collect 𝑀 roll-outs for the imitation learning projection. Setting the 3 Other mirror descent-style analyses, such as in (Sun et al., 2018b), lead to accumulation of errors over the rounds of learning 𝑇. One key difference is that we are leveraging the assumption of convexity of 𝐽 in the (infinite-dimensional) function space representation. 78 c learning rate 𝜂 “ have that: b ` 1 1 𝐻 `𝑀 ` 𝜎2 𝑇 logp𝑇 {𝛿q ˘ , after 𝑇 rounds of the algorithm, we 𝑀 𝑇 1ÿ 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď 𝑇 𝑡 “1 ¨ d ˛ ˜ c ¸ c logp𝑇{𝛿q logp𝑇 𝑘{𝛿q 𝐴𝐻 logp𝑇 𝑘{𝛿q 1 𝐻 ‚` 𝑂 𝜎 𝑂 ˝𝜎 ` ` ` 𝑇 𝑀 𝑀 𝑚 𝑚 holds with probability at least 1 ´ 𝛿, with 𝐻 being the task horizon, 𝐴 the cardinality of action space, 𝜎 2 the variance of policy gradient estimates, and 𝑘 the dimension Π’s parameterization. The expanded result and proof are included in Appendix D.1. The proof leverages previous analysis from DAgger (Ross et al., 2011a) and the finite sample analysis of vanilla policy gradient algorithm (Kakade et al., 2003). The finite-sample regret bound scales linearly with the standard deviation 𝜎 of the gradient estimate, while the bias, which is the very last component of the RHS, scales linearly with the task horizon 𝐻. Note that the standard deviation 𝜎 can be exponential in task horizon 𝐻 in the worst case (Kakade et al., 2003), and so it is important to have practical implementation strategies to reduce the variance of the Update operation. While conducted in a stylized setting, this analysis provides insight in the relative tradeoffs of spending effort in obtaining more accurate projections versus more reliable gradient estimates. Closing the gap between UpdateH and UpdateF Our functional mirror descent analysis rests on taking gradients in H : UpdateH p𝜋q involves estimating ∇H 𝐽p𝜋q in the H space. On the other hand, Algorithm 6 performs UpdateF p𝜋q only in the neural policy space F . In either case, although 𝐽p𝜋q may be differentiable in the non-parametric ambient policy space, it may not be possible to obtain a differentiable parametric programmatic representation in Π. In this section, we discuss theoretical motivations to addressing a practical issue: How do we define and approximate the gradient ∇H 𝐽p𝜋q under a parametric representation? To our knowledge, we are the first to consider such a theoretical question for reinforcement learning. Defining a consistent approximation of ∇H 𝐽p𝜋q. The idea in UpdateF p𝜋q (Line 8 of Alg. 5) is to approximate ∇H 𝐽p𝜋q by ∇F 𝐽p 𝑓 q, which has a differentiable 79 representation, at some 𝑓 close to 𝜋 (under the norm). Under appropriate conditions on F , we show that this approximation is valid. Proposition 5.4.3. Assume that (i) 𝐽 is Fréchet differentiable on H , (ii) 𝐽 is also differentiable on the restricted subspace F , and (iii) F is dense in H (i.e., the s “ H ). Then for any fixed policy 𝜋 P Π, define a sequence of policies closure F 𝑓 𝑘 P F , 𝑘 “ 1, 2, . . .), that converges to 𝜋: lim 𝑘 Ñ8 } 𝑓 𝑘 ´ 𝜋} “ 0. We then have lim 𝑘 Ñ8 }∇F 𝐽p 𝑓 𝑘 q ´ ∇H 𝐽p𝜋q}˚ “ 0. Since the Fréchet gradient is unique in the ambient space H , @𝑘 we have ∇H 𝐽p 𝑓 𝑘 q “ ∇F 𝐽p 𝑓 𝑘 q Ñ ∇H 𝐽p𝜋q as 𝑘 Ñ 8 (by Proposition 5.4.3). We thus have an asymptotically unbiased approximation of ∇H 𝐽p𝜋q via differentiable space F as: ∇F 𝐽p𝜋q fi ∇H 𝐽p𝜋q fi lim 𝑘 Ñ8 ∇F 𝐽p 𝑓 𝑘 q.4 Connecting to the result from Theorem 5.4.1, let 𝜎 2 be an upper bound on the policy gradient estimates in the neural policy class F , under an asymptotically unbiased of ∇H 𝐽p𝜋q, ” ř ı approximation ´ b ¯ the expected regret 𝑇 1 1 bound becomes E 𝑇 𝑡 “1 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝜎 𝑇 ` 𝜖 . Bias-variance considerations of UpdateF p𝜋q To further theoretically motivate a practical strategy for UpdateF p𝜋q in Algorithm 6, we utilize an equivalent proximal perspective of mirror descent (Beck and Teboulle, 2003), where UpdateH p𝜋q is equivalent to solving for ℎ1 “ arg minℎPH 𝜂x∇H 𝐽p𝜋q, ℎy ` 𝐷 𝑅 pℎ, 𝜋q. Proposition 5.4.4 (Minimizing a relaxed objective). For a fixed programmatic policy 𝜋, with sufficiently small constant 𝜆 P p0, 1q, we have that ` ˘ min 𝜂x∇H 𝐽p𝜋q, ℎqy ` 𝐷 𝑅 pℎ, 𝜋q ď min 𝐽 𝜋 ` 𝜆 𝑓 ´ 𝐽p𝜋q ` x∇𝐽p𝜋q, 𝜋y ℎPH 𝑓 PF (5.3) Thus, a relaxed UpdateH step is obtained by minimizing the RHS of (5.3), i.e., minimizing 𝐽p𝜋 ` 𝜆 𝑓 q over 𝑓 P F . Each gradient descent update step is now 𝑓 1 “ 𝑓 ´ 𝜂𝜆∇F 𝐽p𝜋𝑡 `𝜆 𝑓 q, corresponding to Line 5 of Algorithm 6. For fixed 𝜋 and small 𝜆, this relaxed optimization problem becomes regularized policy optimization over F , which is significantly easier. Functional regularization in policy space around a fixed prior controller 𝜋 has demonstrated significant reduction in the variance of gradient estimate (Cheng et al., 2019e), at the expense of some bias. The below expected regret bound summarizes the impact of this increased bias and reduced variance, with details included in Appendix D.1. 4 We do not assume 𝐽p𝜋q to be differentiable when restricting to the policy subspace Π, i.e., ∇Π 𝐽p𝜋q may not exist under policy parameterization of Π. 80 Proposition 5.4.5 (Bias-variance characterization of UpdateF ). Assuming 𝐽pℎq is 𝐿-strongly smooth over H , i.e., ∇H 𝐽pℎq is 𝐿-Lipschitz continuous, approximating per Alg. 6 leads to¯ the expected regret bound: b ” ř UpdateHı by Update𝐹 ´ E 𝑇1 𝑇𝑡“1 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝜆𝜎 𝑇1 ` 𝜖 ` 𝜆2 𝐿 2 . Compared to the idealized unbiased approximation in Proposition 5.4.3, the introduced bias here is related to the inherent smoothness property of cost functional 𝐽pℎq over the joint policy class H , i.e., how close 𝐽p𝜋 ` 𝜆 𝑓 q is to its linear underapproximation 𝐽p𝜋q ` x∇H 𝐽p𝜋q, 𝜆 𝑓 y around 𝜋. 5.5 Experiments We demonstrate the effectiveness of Propel in synthesizing programmatic controllers in three continuous control environments. For brevity and focus, this section primarily focuses on Torcs5, a challenging race car simulator environment (Wymann et al., 2014). Empirical results on two additional classic control tasks, Mountain-Car and Pendulum, are provided in Appendix D.2; those results follow similar trends as the ones described for Torcs below, and further validate the convergence analysis of Propel. Experimental Setup. We evaluate over five distinct tracks in the Torcs simulator. The difficulty of a track can be characterized by three properties; track length, track width, and number of turns. Our suite of tracks provides environments with varying levels of difficulty for the learning algorithm. The performance of a policy in the Torcs simulator is measured by the lap time achieved on the track. To calculate the lap time, the policies are allowed to complete a three-lap race, and we record the best lap time during this race. We perform the experiments with twenty-five random seeds and report the median lap time over these twenty-five trials. Some of the policies crash the car before completing a lap on certain tracks, even after training for 600 episodes. Such crashes are recorded as a lap time of infinity while calculating the median. If the policy crashes for more than half the seeds, this is reported as Cr in Tables 5.1 & 5.2. We choose to report the median because taking the crash timing as infinity, or an arbitrarily large constant, heavily skews other common measures such as the mean. Baselines. Among recent state-of-the-art approaches to learning programmatic policies are Ndps (Verma et al., 2018) for high-level language policies, and Viper 5 The code for the Torcs experiments can be found at: https://bitbucket.org/averma8053/propel Lap Time Improvement 81 G-Track E-Road Aalborg Ruudskogen Alpine-2 200 150 100 50 0 0 1 2 3 Iterations 4 5 Figure 5.4: Median lap-time improvements during multiple iterations of IppgProgram over 25 random seeds. Number of Crashes 600 Max Episodes 500 400 DDPG PROPEL-Prog 300 200 100 0 1 2 3 Track ID 4 5 Figure 5.5: Median number of crashes during training of Ddpg and IppgProgram over 25 random seeds. (Bastani et al., 2018) for learning tree-based policies. Both Ndps and Viper rely on imitating a fixed (pre-trained) neural policy oracle, and can be viewed as degenerate versions of Propel that only run Lines 4-6 in Algorithm 5. We present two Propel analogues to Ndps and Viper: (i) IppgProgram: Propel using the high-level language of Figure 5.1 as the class of programmatic policies, similar to Ndps. (ii) IppgTree: Propel using regression trees, similar to Viper. We also report results for Prior, which is a (sub-optimal) PID controller that is also used as the initial policy in Propel. In addition, to study generalization ability as well as safety behavior during training, we also include Ddpg, a neural policy learned using the Deep Deterministic Policy Gradients (Lillicrap et al., 2015) algorithm, with 600 episodes of training for each track. In principle, Propel and its analysis can accommodate different policy gradient subroutines. However, in the Torcs domain, other policy gradient algorithms such as PPO and TRPO failed to learn policies that are able to complete the considered tracks. We thus focus on Ddpg as our main policy gradient component. Evaluating Performance. Table 5.1 shows the performance on the considered Torcs tracks. We see that IppgProgram and IppgTree consistently outperform the Ndps (Verma et al., 2018) and Viper (Bastani et al., 2018) baselines, respectively. 82 Table 5.1: Performance results in Torcs over 25 random seeds. Each entry is formatted as Lap-time / Crash-ratio, reporting median lap time in seconds over all the seeds (lower is better) and ratio of seeds that result in crashes (lower is better). A lap time of Cr indicates the agent crashed and could not complete a lap for more than half the seeds. Length Prior Ddpg Ndps Viper IppgProgram IppgTree G-Track 3186m E-Road 3260m Aalborg 2588m Ruudskogen 3274m Alpine-2 3774m 312.92 / 0.0 78.82 / 0.24 108.25 / 0.24 83.60 / 0.24 93.67 / 0.04 78.33 / 0.04 322.59 / 0.0 89.71 / 0.28 126.80 / 0.28 87.53 / 0.28 119.17 / 0.04 79.39 / 0.04 244.19 / 0.0 101.06 / 0.40 163.25 / 0.40 110.57 / 0.40 147.28 / 0.12 109.83 / 0.16 340.29 / 0.0 Cr / 0.68 Cr / 0.68 Cr / 0.68 124.58 / 0.16 118.80 / 0.24 402.89 / 0.0 Cr / 0.92 Cr / 0.92 Cr / 0.92 256.59 / 0.16 236.01 / 0.36 While Ddpg outperforms Propel on some tracks, its volatility causes it to be unable to learn in some environments, and hence to crash the majority of the time. Figure 5.4 shows the consistent improvements made over the prior by IppgProgram, over the iterations of the Propel algorithm. Appendix D.2 contains similar results achieved on the two classic control tasks, MountainCar and Pendulum. Figure 5.5 shows that, compared to Ddpg, our approach suffers far fewer crashes while training in Torcs. Evaluating Generalization. To compare the ability of the controllers to perform on tracks not seen during training, we executed the learned policies on all the other tracks (Table 5.2). We observe that Ddpg crashes significantly more often than IppgProgram. This demonstrates the generalizability of the policies returned by Propel. Generalization results for the IppgTree policy are given in the appendix. In general, IppgTree policies are more generalizable than Ddpg but less than IppgProgram. On an absolute level, the generalization ability of Propel still leaves much room for improvement, which is an interesting direction for future work. Verifiability of Policies. As shown in prior work (Bastani et al., 2018; Verma et al., 2018), parsimonious programmatic policies are more amenable to formal verification than neural policies. Unsurprisingly, the policies generated by IppgTree and IppgProgram are easier to verify than Ddpg policies. As a concrete example, we verified a smoothness property of the IppgProgram policy using the Z3 SMT-solver (de Moura and Bjørner, 2008) (more details in Appendix D.2). The verification terminated in 0.49 seconds. Initialization. In principle, Propel can be initialized with a random program, or a random policy trained using Ddpg. In practice, the performance of Propel 83 Table 5.2: Generalization results in Torcs, where rows are training and columns are testing tracks. Each entry is formatted as IppgProgram / DDPG, and the number reported is the median lap time in seconds over all the seeds (lower is better). Cr indicates the agent crashed and could not complete a lap for more than half the seeds. G-Track E-Road Aalborg Ruudskogen Alpine-2 G-Track E-Road Aalborg Ruudskogen Alpine-2 102 / 92 201 / 91 131 / Cr 222 / Cr 124 / Cr 228 / Cr 135 / Cr 231 / Cr Cr / Cr Cr / Cr Cr / Cr 184 / Cr Cr / Cr Cr / Cr 217 / Cr Cr / Cr Cr / Cr Cr / Cr Cr / Cr Cr / Cr - depends to a certain degree on the stability of the policy gradient procedure, which is Ddpg in our experiments. Unfortunately, Ddpg often exhibits high variance across trials and fares poorly in challenging RL domains. Specifically, in our Torcs experiments, Ddpg fails on a number of tracks (similar phenomena have been reported in previous work that experiments on similar continuous control domains (Henderson et al., 2018; Cheng et al., 2019e; Verma et al., 2018)). Agents obtained by initializing Propel with neural policies obtained via Ddpg also fail on multiple tracks. Their performance over the five tracks is reported in Appendix D.2. In contrast, Propel can often finish the challenging tracks when initialized with a very simple hand-crafted programmatic prior. 5.6 Related Work Program Synthesis. Program synthesis is the problem of automatically searching for a program within a language that fits a given specification (Gulwani et al., 2017). Recent approaches to the problem have leveraged symbolic knowledge about program structure (Feser et al., 2015), satisfiability solvers (Solar-Lezama et al., 2006; Jha et al., 2010), and meta-learning techniques (Murali et al., 2018; Parisotto et al., 2016; Devlin et al., 2017; Balog et al., 2017) to generate interesting programs in many domains (Alur et al., 2013; Polozov and Gulwani, 2015; Alur et al., 2017). In most prior work, the specification is a logical constraint on the input/output behavior of the target program. However, there is also a growing body of work that considers program synthesis modulo optimality objectives (Bloem et al., 2009; Chaudhuri et al., 2014; Raychev et al., 2016), often motivated by machine learning tasks (Murali et al., 2018; Valkov et al., 2018; Ellis et al., 2018; Du et al., 2018; Verma et al., 2018; Bastani et al., 2018; Zhu et al., 2019). Synthesis of programs 84 that imitates an oracle has been considered in both the logical (Jha et al., 2010) and the optimization (Verma et al., 2018; Bastani et al., 2018; Zhu et al., 2019) settings. The projection step in Propel builds on this prior work. While our current implementation of this step is entirely symbolic, in principle, the operation can also utilize contemporary techniques for learning policies that guide the synthesis process (Murali et al., 2018; Balog et al., 2017; Si et al., 2019). Constrained Policy Learning. Constrained policy learning has seen increased interest in recent years, largely due to the desire to impose side guarantees such as stability and safety on the policy’s behavior. Broadly, there are two approaches to imposing constraints: specifying constraints as an additional cost function (Achiam et al., 2017; Le et al., 2019b), and explicitly encoding constraints into the policy class (Alshiekh et al., 2018; Le et al., 2016b; Cheng et al., 2019e; Dalal et al., 2018; Berkenkamp et al., 2017). In some cases, these two approaches can be viewed as duals of each other. For instance, recent work that uses control-theoretic policies as a functional regularizer (Le et al., 2016b; Cheng et al., 2019e) can be viewed from the perspective of both regularization (additional cost) and an explicitly constrained policy class (a specific mix of neural and control-theoretic policies). We build upon this perspective to develop the gradient update step in our approach. RL using Imitation Learning. There are two ways to utilize imitation learning subroutines within RL. First, one can leverage limited-access or sub-optimal experts to speed up learning (Ross and Bagnell, 2014; Cheng et al., 2019a; Chang et al., 2015; Sun et al., 2018a). Second, one can learn over two policy classes (or one policy and one model class) to achieve accelerated learning compared to using only one policy class (Montgomery and Levine, 2016; Cheng et al., 2019c; Sun et al., 2018b; Cheng et al., 2019b). Our approach has some stylistic similarities to previous efforts (Montgomery and Levine, 2016; Sun et al., 2018b) that use a richer policy space to search for improvements before re-training the primary policy to imitate the richer policy. One key difference is that our primary policy is programmatic and potentially non-differentiable. A second key difference is that our theoretical framework takes a functional gradient descent perspective — it would be interesting to carefully compare with previous analysis techniques to find a unifying framework. RL with Mirror Descent. The mirror descent framework has previously used to analyze and design RL algorithms. For example, Thomas et al. (Thomas et al., 2013) and Mahadevan and Liu (Mahadevan and Liu, 2012) use composite objective mirror descent, or Comid (Duchi et al., 2010), which allows incorporating adap- 85 tive regularizers into gradient updates, thus offering connections to either natural gradient RL (Thomas et al., 2013) or sparsity inducing RL algorithms (Mahadevan and Liu, 2012). Unlike the approach in this section, these prior approaches perform projection into the same native, differentiable representation. Also, the analyses in these prior work do not consider errors introduced by hybrid representations and approximate projection operators. However, one can potentially extend our approach with versions of mirror descent, e.g., Comid, that were considered in these efforts. 5.7 Conclusion and Future Work We have presented Propel, a meta-algorithm based on mirror descent, program synthesis, and imitation learning, for programmatic reinforcement learning (PRl). We have presented theoretical convergence results for Propel, developing novel analyses to characterize approximate projections and biased gradients within the mirror descent framework. We also validated Propel empirically, and show that it can discover interpretable, verifiable, generalizable, performant policies and significantly outperform the state of the art in PRl. The central idea of Propel is the use of imitation learning and combinatorial methods in implementing a projection operation for mirror descent, with the goal of optimization in a functional space that lacks gradients. While we have developed Propel in an RL setting, this idea is not restricted to RL or even sequential decision making. Future work will seek to exploit this insight in other machine learning and program synthesis settings. 86 Chapter 6 HIERARCHICAL IMITATION AND REINFORCEMENT LEARNING Summary. We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the hierarchical structure of the underlying problem to integrate different modes of expert interaction. Our framework can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels, leading to dramatic reductions in both expert effort and cost of exploration. Using longhorizon benchmarks, including Montezuma’s Revenge, we demonstrate that our approach can learn significantly faster than hierarchical RL, and be significantly more label-efficient than standard IL. We also theoretically analyze labeling cost for certain instantiations of our framework. 6.1 Introduction Learning good agent behavior from reward signals alone—the goal of reinforcement learning (RL)—is particularly difficult when the planning horizon is long and rewards are sparse. One successful method for dealing with such long horizons is imitation learning (IL) (Abbeel and Ng, 2004; Daumé III et al., 2009; Ross et al., 2011b; Ho and Ermon, 2016), in which the agent learns by watching and possibly querying an expert. One limitation of existing imitation learning approaches is that they may require a large amount of demonstration data in long-horizon problems. The central question we address in this chapter of the thesis is: when experts are available, how can we most effectively leverage their feedback? A common strategy to improve sample efficiency in RL over long time horizons is to exploit hierarchical structure of the problem (Sutton et al., 1998, 1999; Kulkarni et al., 2016; Dayan and Hinton, 1993; Vezhnevets et al., 2017; Dietterich, 2000). Our approach leverages hierarchical structure in imitation learning. We study the case where the underlying problem is hierarchical, and subtasks can be easily elicited from an expert. Our key design principle is an algorithmic framework called hierarchical guidance, in which feedback (labels) from the high-level expert is used to focus (guide) the low-level 87 learner. The high-level expert ensures that low-level learning only occurs when necessary (when subtasks have not been mastered) and only over relevant parts of the state space. This differs from a naïve hierarchical approach which merely gives a subtask decomposition. Focusing on relevant parts of the state space speeds up learning (improves sample efficiency), while omitting feedback on the already mastered subtasks reduces expert effort (improves label efficiency). We begin by formalizing the problem of hierarchical imitation learning (section 6.3) and carefully separate out cost structures that naturally arise when the expert provides feedback at multiple levels of abstraction. We first apply hierarchical guidance to IL, derive hierarchically guided variants of behavior cloning and DAgger (Ross et al., 2011b), and theoretically analyze the benefits (section 6.4). We next apply hierarchical guidance to the hybrid setting with high-level IL and low-level RL (section 6.5). This architecture is particularly suitable in settings where we have access to high-level semantic knowledge, the subtask horizon is sufficiently short, but the low-level expert is too costly or unavailable. We demonstrate the efficacy of our approaches on a simple but extremely challenging maze domain, and on Montezuma’s Revenge (section 6.6). Our experiments show that incorporating a modest amount of expert feedback can lead to dramatic improvements in performance compared to pure hierarchical RL.1 6.2 Related Work in Imitation and Reinforcement Learning For brevity, we provide here a short overview of related work, and defer to Appendix E.3 for additional discussion. Imitation Learning. One can broadly dichotomize IL into passive collection of demonstrations (behavioral cloning) versus active collection of demonstrations. The former setting (Abbeel and Ng, 2004; Ziebart et al., 2008; Syed and Schapire, 2008; Ho and Ermon, 2016) assumes that demonstrations are collected a priori and the goal of IL is to find a policy that mimics the demonstrations. The latter setting (Daumé III et al., 2009; Ross et al., 2011b; Ross and Bagnell, 2014; Chang et al., 2015; Sun et al., 2017) assumes an interactive expert that provides demonstrations in response to actions taken by the current policy. We explore extension of both approaches into hierarchical settings. Hierarchical Reinforcement Learning. Several RL approaches to learning hier1 Code and experimental setups are available at https://sites.google.com/view/ hierarchical-il-rl 88 archical policies have been explored, foremost among them the options framework (Sutton et al., 1998, 1999; Fruit and Lazaric, 2017). It is often assumed that a useful set of options are fully defined a priori, and (semi-Markov) planning and learning only occurs at the higher level. In comparison, our agent does not have direct access to policies that accomplish such subgoals and has to learn them via expert or reinforcement feedback. The closest hierarchical RL work to ours is that of (Kulkarni et al., 2016), which uses a similar hierarchical structure, but no high-level expert and hence no hierarchical guidance. Combining Reinforcement and Imitation Learning. The idea of combining IL and RL is not new (Nair et al., 2017; Hester et al., 2018). However, previous work focuses on flat policy classes that use IL as a “pre-training” step (e.g., by prepopulating the replay buffer with demonstrations). In contrast, we consider feedback at multiple levels for a hierarchical policy class, with different levels potentially receiving different types of feedback (i.e., imitation at one level and reinforcement at the other). Somewhat related to our hierarchical expert supervision is the approach of (Andreas et al., 2017), which assumes access to symbolic descriptions of subgoals, without knowing what those symbols mean or how to execute them. Previous literature has not focused much on comparisons of sample complexity between IL and RL, with the exception of the recent work of (Sun et al., 2017). 6.3 Hierarchical Formalism For simplicity, we consider environments with a natural two-level hierarchy; the hi level corresponds to choosing subtasks, and the lo level corresponds to executing those subtasks. For instance, an agent’s overall goal may be to leave a building. At the hi level, the agent may first choose the subtask “go to the elevator,” then “take the elevator down,” and finally “walk out.” Each of these subtasks needs to be executed at the lo level by actually navigating the environment, pressing buttons on the elevator, etc.2 Subtasks, which we also call subgoals, are denoted as 𝑔 P G, and the primitive actions are denoted as 𝑎 P A. An agent (also referred to as learner) acts by iteratively choosing a subgoal 𝑔, carrying it out by executing a sequence of actions 𝑎 2 An important real-world application is in goal-oriented dialogue systems. For instance, a chatbot assisting a user with reservation and booking for flights and hotels (Peng et al., 2017; El Asri et al., 2017) needs to navigate through multiple turns of conversation. The chatbot developer designs the hierarchy of subtasks, such as ask_user_ goal, ask_dates, offer_ flights, confirm, etc. Each subtask consists of several turns of conversation. Typically a global state tracker exists alongside the hierarchical dialogue policy to ensure that cross-subtask constraints are satisfied. 89 until completion, and then picking a new subgoal. The agent’s choices can depend on an observed state 𝑠 P S.3 We assume that the horizon at the hi level is 𝐻hi , i.e., a trajectory uses at most 𝐻hi subgoals, and the horizon at the lo level is 𝐻lo , i.e., after at most 𝐻lo primitive actions, the agent either accomplishes the subgoal or needs to decide on a new subgoal. The total number of primitive actions in a trajectory is thus at most 𝐻full B 𝐻hi 𝐻lo . The hierarchical learning problem is to simultaneously learn a hi-level policy 𝜇 : S Ñ G, called the meta-controller, as well as the subgoal policies 𝜋𝑔 : S Ñ A for each 𝑔 P G, called subpolicies. The aim of the learner is to achieve a high reward when its meta-controller and subpolicies are run together. For each subgoal 𝑔, we also have a (possibly learned) termination function 𝛽𝑔 : S Ñ tTrue, Falseu, which terminates the execution of 𝜋𝑔 . The hierarchical agent behaves as follows: 1: for ℎhi “ 1 . . . 𝐻hi do observe state 𝑠 and choose subgoal 𝑔 Ð 𝜇p𝑠q 3: for ℎlo “ 1 . . . 𝐻lo do 4: observe state 𝑠 5: if 𝛽𝑔 p𝑠q then break 6: choose action 𝑎 Ð 𝜋𝑔 p𝑠q 7: end for 8: end for 2: The execution of each subpolicy 𝜋𝑔 generates a lo-level trajectory 𝜏 “ p𝑠1 , 𝑎 1 , . . . , 𝑠 𝐻 , 𝑎 𝐻 , 𝑠 𝐻 `1 q with 𝐻 ď 𝐻lo .4 The overall behavior results in a hierarchical trajectory 𝜎 “ p𝑠1 , 𝑔1 , 𝜏1 , 𝑠2 , 𝑔2 , 𝜏2 , . . . q, where the last state of each lo-level trajectory 𝜏ℎ coincides with the next state 𝑠 ℎ`1 in 𝜎 and the first state of the next lo-level trajectory 𝜏ℎ`1 . The subsequence of 𝜎 which excludes the lo-level trajectories 𝜏ℎ is called the hi-level trajectory, 𝜏hi B p𝑠1 , 𝑔1 , 𝑠2 , 𝑔2 , . . . q. Finally, the full trajectory, 𝜏full , is the concatenation of all the lo-level trajectories. We assume access to an expert, endowed with a meta-controller 𝜇‹ , subpolicies 𝜋𝑔‹ , and termination functions 𝛽𝑔‹ , who can provide one or several types of supervision: • HierDemop𝑠q: hierarchical demonstration. The expert executes its hierarchical policy starting from 𝑠 and returns the resulting hierarchical trajectory 3 While we use the term state for simplicity, we do not require the environment to be fully observable or Markovian. 4 The trajectory might optionally include a reward signal after each primitive action, which might either come from the environment, or be a pseudo-reward as we will see in Section 6.5. 90 𝜎 ‹ “ p𝑠1‹ , 𝑔1‹ , 𝜏1‹ , 𝑠2‹ , 𝑔2‹ , 𝜏2‹ , . . . q, where 𝑠1‹ “ 𝑠. • Labelhi p𝜏hi q: hi-level labeling. The expert provides a good next subgoal at each state of a given hi-level trajectory 𝜏hi “ p𝑠1 , 𝑔1 , 𝑠2 , 𝑔2 , . . . q, yielding a labeled data set tp𝑠1 , 𝑔1‹ q, p𝑠2 , 𝑔2‹ q, . . . u. • Labello p𝜏; 𝑔q: lo-level labeling. The expert provides a good next primitive action towards a given subgoal 𝑔 at each state of a given lo-level trajectory 𝜏 “ p𝑠1 , 𝑎 1 , 𝑠2 , 𝑎 2 , . . . q, yielding a labeled data set tp𝑠1 , 𝑎 ‹1 q, p𝑠2 , 𝑎 ‹2 q, . . . u. • Inspectlo p𝜏; 𝑔q: lo-level inspection. Instead of annotating every state of a trajectory with a good action, the expert only verifies whether a subgoal 𝑔 was accomplished, returning either Pass or Fail. • Labelfull p𝜏full q: full labeling. The expert labels the agent’s full trajectory 𝜏full “ p𝑠1 , 𝑎 1 , 𝑠2 , 𝑎 2 , . . . q, from start to finish, ignoring hierarchical structure, yielding a labeled data set tp𝑠1 , 𝑎 ‹1 q, p𝑠2 , 𝑎 ‹2 q, . . . u. • Inspectfull p𝜏full q: full inspection. The expert verifies whether the agent’s overall goal was accomplished, returning either Pass or Fail. When the agent learns not only the subpolicies 𝜋𝑔 , but also termination functions 𝛽𝑔 , then Labello also returns good termination values 𝜔‹ P tTrue, Falseu for each state of 𝜏 “ p𝑠1 , 𝑎 1 . . . q, yielding a data set tp𝑠1 , 𝑎 ‹1 , 𝜔‹1 q, . . . u. Although HierDemo and Label can be both generated by the expert’s hierarchical policy p𝜇‹ , t𝜋𝑔‹ uq, they differ in the mode of expert interaction. HierDemo returns a hierarchical trajectory executed by the expert, as required for passive IL, and enables a hierarchical version of behavioral cloning (Abbeel and Ng, 2004; Syed and Schapire, 2008). Label operations provide labels with respect to the learning agent’s trajectories, as required for interactive IL. Labelfull is the standard query used in prior work on learning flat policies (Daumé III et al., 2009; Ross et al., 2011b), and Labelhi and Labello are its hierarchical extensions. Inspect operations are newly introduced in this work, and form a cornerstone of our interactive hierarchical guidance protocol that enables substantial savings in label efficiency. They can be viewed as “lazy” versions of the corresponding Label operations, requiring less effort. Our underlying assumption is that if the given hierarchical trajectory 𝜎 “ tp𝑠 ℎ , 𝑔 ℎ , 𝜏ℎ qu agrees with the expert on hi level, i.e., 𝑔 ℎ “ 𝜇‹ p𝑠 ℎ q, and lo-level trajectories pass the inspection, i.e., Inspectlo p𝜏ℎ ; 𝑔 ℎ q “ Pass, then the resulting full trajectory must also pass the full inspection, Inspectfull p𝜏full q “ Pass. 91 Algorithm 8 Hierarchical Behavioral Cloning (h-BC) 1: Initialize data buffers Dhi Ð H and D𝑔 Ð H, 𝑔 P G 2: for 𝑡 “ 1, . . . , 𝑇 do 3: Get a new environment instance with start state 𝑠 4: 𝜎 ‹ Ð HierDemop𝑠q 5: for all p𝑠‹ℎ , 𝑔 ‹ℎ , 𝜏ℎ‹ q P 𝜎 ‹ do 6: Append D𝑔‹ℎ Ð D𝑔‹ℎ Y 𝜏ℎ‹ 7: Append Dhi Ð Dhi Y tp𝑠‹ℎ , 𝑔 ‹ℎ qu 8: end for 9: end for 10: Train subpolicies 𝜋𝑔 Ð Trainp𝜋𝑔 , D𝑔 q for all 𝑔 11: Train meta-controller 𝜇 Ð Trainp𝜇, Dhi q This means that a hierarchical policy need not always agree with the expert’s execution at lo level to succeed in the overall task. Besides algorithmic reasons, the motivation for separating the types of feedback is that different expert queries will typically require different amount of effort, which L , 𝐶 L and we refer to as cost. We assume the costs of the Label operations are 𝐶hi lo L I I 𝐶full , the costs of each Inspect operation are 𝐶lo and 𝐶full . In many settings, lo-level inspection will require significantly less effort than lo-level labeling, i.e., I ! 𝐶 L . For instance, identifying if a robot has successfully navigated to the 𝐶lo lo elevator is presumably much easier than labeling an entire path to the elevator. One reasonable cost model, natural for the environments in our experiments, is to assume that Inspect operations take time 𝑂p1q and work by checking the final state of the trajectory, whereas Label operations take time proportional to the trajectory length, which is 𝑂p𝐻hi q, 𝑂p𝐻lo q and 𝑂p𝐻hi 𝐻lo q for our three Label operations. 6.4 Hierarchically Guided Imitation Learning Hierarchical guidance is an algorithmic design principle in which the feedback from high-level expert guides the low-level learner in two different ways: (i) the high-level expert ensures that low-level expert is only queried when necessary (when the subtasks have not been mastered yet), and (ii) low-level learning is limited to the relevant parts of the state space. We instantiate this framework first within passive learning from demonstrations, obtaining hierarchical behavioral cloning (Algorithm 8), and then within interactive imitation learning, obtaining hierarchically guided DAgger (Algorithm 9), our best-performing algorithm. 92 Algorithm 9 Hierarchically Guided DAgger (hg-DAgger) 1: Initialize data buffers Dhi Ð H and D𝑔 Ð H, 𝑔 P G 2: Run Hierarchical Behavioral Cloning (Algorithm 8) up to 𝑡 “ 𝑇warm-start 3: for 𝑡 “ 𝑇warm-start ` 1, . . . , 𝑇 do 4: Get a new environment instance with start state 𝑠 5: Initialize 𝜎 Ð H 6: repeat 7: 𝑔 Ð 𝜇p𝑠q 8: Execute 𝜋𝑔 , obtain lo-level trajectory 𝜏 9: Append p𝑠, 𝑔, 𝜏q to 𝜎 10: 𝑠 Ð the last state in 𝜏 11: until end of episode 12: Extract 𝜏full and 𝜏hi from 𝜎 13: if Inspectfull p𝜏full q “ Fail then 14: D ‹ Ð Labelhi p𝜏hi q 15: Process p𝑠 ℎ , 𝑔 ℎ , 𝜏ℎ q P 𝜎 in sequence as long as 𝑔 ℎ agrees with the expert’s choice 𝑔 ‹ℎ in D ‹ : 16: if Inspectp𝜏ℎ ; 𝑔 ℎ q “ Fail then 17: Append D𝑔ℎ Ð D𝑔ℎ Y Labello p𝜏ℎ ; 𝑔 ℎ q 18: break 19: end if 20: Append Dhi Ð Dhi Y D ‹ 21: end if 22: Update subpolicies 𝜋𝑔 Ð Trainp𝜋𝑔 , D𝑔 q for all 𝑔 23: Update meta-controller 𝜇 Ð Trainp𝜇, Dhi q 24: end for Hierarchical Behavioral Cloning (h-BC) We consider a natural extension of behavioral cloning to the hierarchical setting (Algorithm 8). The expert provides a set of hierarchical demonstrations 𝜎 ‹ , each consisting of lo-level trajectories 𝜏ℎ‹ “ tp𝑠ℓ‹ , 𝑎 ℓ‹ quℓ𝐻“lo1 as well as a hi-level trajectory ‹ “ tp𝑠 ‹ , 𝑔 ‹ qu𝐻hi . We then run Train (lines 8–9) to find the subpolicies 𝜋 𝜏hi 𝑔 ℎ ℎ ℎ “1 ‹ ‹ ‹ ‹ that best predict 𝑎 ℓ from 𝑠ℓ , and meta-controller 𝜇 that best predicts 𝑔 ℎ from 𝑠 ℎ , respectively. Train can generally be any supervised learning subroutine, such as stochastic optimization for neural networks or some batch training procedure. When termination functions 𝛽𝑔 need to be learned as part of the hierarchical policy, the labels 𝜔‹𝑔 will be provided by the expert as part of 𝜏ℎ‹ “ tp𝑠ℓ‹ , 𝑎 ℓ‹ , 𝜔ℓ‹ qu.5 In this setting, hierarchical guidance is automatic, because subpolicy demonstrations only occur in relevant parts of the state space. 5 In our hierarchical imitation learning experiments, the termination functions are all learned. Formally, the termination signal 𝜔𝑔 , can be viewed as part of an augmented action at lo level. 93 Hierarchically Guided DAgger (hg-DAgger) Passive IL, e.g., behavioral cloning, suffers from the distribution mismatch between the learning and execution distributions. This mismatch is addressed by interactive IL algorithms, such as SEARN (Daumé III et al., 2009) and DAgger (Ross et al., 2011b), where the expert provides correct actions along the learner’s trajectories through the operation Labelfull . A naïve hierarchical implementation would provide correct labels along the entire hierarchical trajectory via Labelhi and Labello . We next show how to use hierarchical guidance to decrease lo-level expert costs. We leverage two hi-level query types: Inspectlo and Labelhi . We use Inspectlo to verify whether the subtasks are successfully completed and Labelhi to check whether we are staying in the relevant part of the state space. The details are presented in Algorithm 9, which uses DAgger as the learner on both levels, but the scheme can be adapted to other interactive imitation learners. In each episode, the learner executes the hierarchical policy, including choosing a subgoal (line 7), executing the lo-level trajectories, i.e., rolling out the subpolicy 𝜋𝑔 for the chosen subgoal, and terminating the execution according to 𝛽𝑔 (line 8). Expert only provides feedback when the agent fails to execute the entire task, as verified by Inspectfull (line 13). When Inspectfull fails, the expert first labels the correct subgoals via Labelhi (line 14), and only performs lo-level labeling as long as the learner’s meta-controller chooses the correct subgoal 𝑔 ℎ (line 15), but its subpolicy fails (i.e., when Inspectlo on line 16 fails). Since all the preceding subgoals were chosen and executed correctly, and the current subgoal is also correct, lo-level learning is in the “relevant” part of the state space. However, since the subpolicy execution failed, its learning has not been mastered yet. We next analyze the savings in expert cost that result from hierarchical guidance. Theoretical Analysis. We analyze the cost of hg-DAgger in comparison with flat DAgger under somewhat stylized assumptions. We assume that the learner aims to learn the meta-controller 𝜇 from some policy class M, and subpolicies 𝜋𝑔 from some class Πlo . The classes M and Πlo are finite (but possibly exponentially large) and the task is realizable, i.e., the expert’s policies can be found in the corresponding classes: 𝜇‹ P M, and 𝜋𝑔‹ P Πlo , 𝑔 P G. This allows us to use the halving algorithm (Shalev-Shwartz et al., 2012) as the online learner on both levels. (The implementation of our algorithm does not require these assumptions.) The halving algorithm maintains a version space over policies, acts by a majority decision, and when it makes a mistake, it removes all the erring policies from the 94 version space. In the hierarchical setting, it therefore makes at most log |M | mistakes on the hi level, and at most log |Πlo | mistakes when learning each 𝜋𝑔 . The mistake bounds can be further used to upper bound the total expert cost in both hg-DAgger and flat DAgger. To enable an apples-to-apples comparison, we assume that the flat DAgger learns over the policy class Πfull “ tp𝜇, t𝜋𝑔 u𝑔PG q : 𝜇 P M, 𝜋𝑔 P Πlo u, but is otherwise oblivious to the hierarchical task structure. The bounds depend on the cost of performing different types of operations, as defined at the end of Section 6.3. We consider a modified version of flat DAgger that first calls Inspectfull , and only requests labels (Labelfull ) if the inspection fails. The proofs are deferred to Appendix E.1. Theorem 6.4.1. Given finite classes M and Πlo and realizable expert policies, the total cost incurred by the expert in hg-DAgger by round 𝑇 is bounded by ` ˘ L I I 𝑇𝐶full ` log2 |M| ` |Gopt | log2 |Πlo | p𝐶hi ` 𝐻hi𝐶lo q ` ˘ L ` |Gopt | log2 |Πlo | 𝐶lo , (6.1) where Gopt Ď G is the set of the subgoals actually used by the expert, Gopt B 𝜇‹ pSq. Theorem 6.4.2. Given the full policy class Πfull “ tp𝜇, t𝜋𝑔 u𝑔PG q : 𝜇 P M, 𝜋𝑔 P Πlo u and a realizable expert policy, the total cost incurred by the expert in flat DAgger by round 𝑇 is bounded by ` ˘ L I 𝑇𝐶full ` log2 |M| ` |G| log2 |Πlo | 𝐶full . (6.2) I Both bounds have the same leading term, 𝑇𝐶full , the cost of full inspection, which is incurred every round and can be viewed as the “cost of monitoring.” In contrast, the remaining terms can be viewed as the “cost of learning” in the two settings, and include terms coming from their respective mistake bounds. The ratio of the cost of hierarchically guided learning to the flat learning is then bounded as L ` 𝐻 𝐶I ` 𝐶L I 𝐶hi Eq. (6.1) ´ 𝑇𝐶full hi lo lo ď , I L Eq. (6.2) ´ 𝑇𝐶full 𝐶full (6.3) where we applied the upper bound |Gopt | ď |G|. The savings thanks to hierarchical guidance depend on the specific costs. Typically, we expect the inspection costs to be 𝑂p1q, if it suffices to check the final state, whereas labeling costs scale linearly with the length of the trajectory. The cost ratio is then 9 𝐻𝐻hihi`𝐻𝐻lolo . Thus, we realize most significant savings if the horizons on each individual level are substantially shorter ? than the overall horizon. In particular, if 𝐻hi “ 𝐻lo “ 𝐻full , the hierarchically 95 Algorithm 10 Hierarchically Guided DAgger / 𝑄-learning input Function pseudop𝑠; 𝑔q providing the pseudo-reward input Predicate terminalp𝑠; 𝑔q indicating the termination of 𝑔 input Annealed exploration probabilities 𝜖 𝑔 ą 0, 𝑔 P G 1: Initialize data buffers Dhi Ð H and D𝑔 Ð H, 𝑔 P G 2: Initialize subgoal 𝑄-functions 𝑄 𝑔 , 𝑔 P G 3: for 𝑡 “ 1, . . . , 𝑇 do 4: Get a new environment instance with start state 𝑠 5: Initialize 𝜎 Ð H 6: repeat 7: 𝑠hi Ð 𝑠, 𝑔 Ð 𝜇p𝑠q and initialize 𝜏 Ð H 8: repeat 9: 𝑎 Ð 𝜖 𝑔 -greedyp𝑄 𝑔 , 𝑠q 10: Execute 𝑎, next state 𝑠˜, 𝑟˜ Ð pseudop𝑠˜; 𝑔q 11: Update 𝑄 𝑔 : a (stochastic) gradient descent step on a minibatch from D𝑔 12: Append p𝑠, 𝑎, 𝑟, ˜ 𝑠˜q to 𝜏 and update 𝑠 Ð 𝑠˜ 13: until terminalp𝑠; 𝑔q 14: Append p𝑠hi , 𝑔, 𝜏q to 𝜎 15: until end of episode 16: Extract 𝜏full and 𝜏hi from 𝜎 17: if Inspectfull p𝜏full q “ Fail then 18: D ‹ Ð Labelhi p𝜏hi q 19: Process p𝑠 ℎ , 𝑔 ℎ , 𝜏ℎ q P 𝜎 in sequence as long as 𝑔 ℎ agrees with the expert’s choice 𝑔 ‹ℎ in D ‹ : 20: Append D𝑔ℎ Ð D𝑔ℎ Y 𝜏ℎ Append Dhi Ð Dhi Y D ‹ 21: else 22: Append D𝑔ℎ Ð D𝑔ℎ Y 𝜏ℎ for all p𝑠 ℎ , 𝑔 ℎ , 𝜏ℎ q P 𝜎 23: end if 24: Update meta-controller 𝜇 Ð Trainp𝜇, Dhi q 25: end for ? guided approach reduces the overall labeling cost by a factor of 𝐻full . More generally, whenever 𝐻full is large, we reduce the costs of learning be at least a constant factor—a significant gain if this is a saving in the effort of a domain expert. 6.5 Hierarchically Guided IL / RL Hierarchical guidance also applies in the hybrid setting with interactive IL on the hi level and RL on the lo level. The hi-level expert provides the hierarchical decomposition, including the pseudo-reward function for each subgoal,6 and is also 6 This is consistent with many hierarchical RL approaches, including options (Sutton et al., 1999), MAXQ (Dietterich, 2000), UVFA (Schaul et al., 2015a) and h-DQN (Kulkarni et al., 2016). 96 able to pick a correct subgoal at each step. Similar to hg-DAgger, the labels from hi-level expert are used not only to train the meta-controller 𝜇, but also to limit the lo-level learning to the relevant part of the state space. In Algorithm 10 we provide the details, with DAgger on hi level and 𝑄-learning on lo level. The scheme can be adapted to other interactive IL and RL algorithms. The learning agent proceeds by rolling in with its meta-controller (line 7). For each selected subgoal 𝑔, the subpolicy 𝜋𝑔 selects and executes primitive actions via the 𝜖-greedy rule (lines 9–10), until some termination condition is met. The agent receives some pseudo-reward, also known as intrinsic reward (Kulkarni et al., 2016) (line 10). Upon termination of the subgoal, agent’s meta-controller 𝜇 chooses another subgoal and the process continues until the end of the episode, where the involvement of the expert begins. As in hg-DAgger, the expert inspects the overall execution of the learner (line 17), and if it is not successful, the expert provides hi-level labels, which are accumulated for training the meta-controller. Hierarchical guidance impacts how the lo-level learners accumulate experience. As long as the meta-controller’s subgoal 𝑔 agrees with the expert’s, the agent’s experience of executing subgoal 𝑔 is added to the experience replay buffer D𝑔 . If the meta-controller selects a “bad” subgoal, the accumulation of experience in the current episode is terminated. This ensures that experience buffers contain only the data from the relevant part of the state space. Algorithm 10 assumes access to a real-valued function pseudop𝑠; 𝑔q, providing the pseudo-reward in state 𝑠 when executing 𝑔, and a predicate terminalp𝑠; 𝑔q, indicating the termination (not necessarily successful) of subgoal 𝑔. This setup is similar to prior work on hierarchical RL (Kulkarni et al., 2016). One natural definition of pseudo-rewards, based on an additional predicate successp𝑠; 𝑔q indicating a successful completion of subgoal 𝑔, is as follows: $ ’ ’ 1 if successp𝑠; 𝑔q ’ & ´1 if successp𝑠; 𝑔q and terminalp𝑠; 𝑔q ’ ’ ’ %´𝜅 otherwise, where 𝜅 ą 0 is a small penalty to encourage short trajectories. The predicates success and terminal are provided by an expert or learnt from supervised or reinforcement feedback. In our experiments, we explicitly provide these predicates to both hg-DAgger/Q as well as the hierarchical RL, giving them advantage over hg-DAgger, which needs to learn when to terminate subpolicies. 97 hg-DAgger (Alg. 2) expert cost per type HI-level labeling LO-level labeling LO-level inspection 10K expert cost 8K 6K 4K 2K 0K episode 0-250 250-500 (success rate 27%) (81%) hg-DAgger/Q (Alg. 3) RL samples vs. expert cost 100% 500-750 (91%) 750-1000 (97%) 40K success rate 30K 25K 60% 20K 40% 15K 20% success rate 10K HI-level expert cost 5K every 5K episodes HI-level expert cost 35K 80% 0% 0K 0K 50K 100K 150K 200K 250K 300K 350K 400K RL samples at LO-level 100% 100% 80% 80% 80% 60% 40% hg-DAgger h-BC flat DAgger flat beh. cloning 20% 0% 0 200 400 600 800 episode (rounds of learning) 1000 60% 40% hg-DAgger h-BC flat DAgger flat beh. cloning 20% 0% 0K 10K 20K 30K 40K 50K 60K expert cost (HI + LO levels) 70K success rate 100% success rate success rate Figure 6.1: Maze navigation. (Left) One sampled environment instance; the agent needs to navigate from bottom right to bottom left. (Middle) Expert cost over time for hg-DAgger; the cost of Label operations equals the length of labeled trajectory, the cost of Inspect operations is 1. (Right) Success rate of hg-DAgger/Q and the hi-level label cost as a function of the number of lo-level RL samples. 60% 40% 20% 0% 0K hg-DAgger flat DAgger 10K 20K 30K 40K 50K 60K expert cost (LO-level) Figure 6.2: Maze navigation: hierarchical versus flat imitation learning. Each episode is followed by a round of training and a round of testing. The success rate is measured over previous 100 test episodes; the expert cost is as in Figure 6.1. (Left) Success rate per episode. (Middle) Success rate versus the expert cost. (Right) Success rate versus the lo-level expert cost. 6.6 Experiments We evaluate the performance of our algorithms on two separate domains: (i) a simple but challenging maze navigation domain and (ii) the Atari game Montezuma’s Revenge. 98 Maze Navigation Domain Task Overview. Figure 6.1 (left) displays a snapshot of the maze navigation domain. In each episode, the agent encounters a new instance of the maze from a large collection of different layouts. Each maze consists of 16 rooms arranged in a 4-by-4 grid, but the openings between the rooms vary from instance to instance as does the initial position of the agent and the target. The agent (white dot) needs to navigate from one corner of the maze to the target marked in yellow. Red cells are obstacles (lava), which the agent needs to avoid for survival. The contextual information the agent receives is the pixel representation of a bird’s-eye view of the environment, including the partial trail (marked in green) indicating the visited locations. Due to a large number of random environment instances, this domain is not solvable with tabular algorithms. Note that rooms are not always connected, and the locations of the hallways are not always in the middle of the wall. Primitive actions include going one step up, down, left or right. In addition, each instance of the environment is designed to ensure that there is a path from initial location to target, and the shortest path takes at least 45 steps (𝐻full “ 100). The agent is penalized with reward ´1 if it runs into lava, which also terminates the episode. The agent only receives positive reward upon stepping on the yellow block. A hierarchical decomposition of the environment corresponds to four possible subgoals of going to the room immediately to the north, south, west, east, and the fifth possible subgoal go_to_target (valid only in the room containing the target). In this setup, 𝐻lo « 5 steps, and 𝐻hi « 10–12 steps. The episode terminates after 100 primitive steps if the agent is unsuccessful. The subpolicies and meta-controller use similar neural network architectures and only differ in the number of action outputs. (Details of network architecture are provided in Appendix E.2.) Hierarchically Guided IL. We first compare our hierarchical IL algorithms with their flat versions. The algorithm performance is measured by success rate, defined as the average rate of successful task completion over the previous 100 test episodes, on random environment instances not used for training. The cost of each Label operation equals the length of the labeled trajectory, and the cost of each Inspect operation equals 1. Both h-BC and hg-DAgger outperform flat imitation learners (6.2,left ). hg-DAgger, in particular, achieves consistently the highest success rate, approaching 100% in fewer than 1000 episodes. 6.2 (left) displays the median as well as the range from minimum to maximum success rate over 5 random executions of the algorithms. 99 Expert cost varies significantly between the two hierarchical algorithms. Figure 6.2 (middle) displays the same success rate, but as a function of the expert cost. hg-DAgger achieves significant savings in expert cost compared to other imitation learning algorithms thanks to a more efficient use of the lo-level expert through hierarchical guidance. Figure 6.1 (middle) shows that hg-DAgger requires most of its lo-level labels early in the training and requests primarily hi-level labels after the subgoals have been mastered. As a result, hg-DAgger requires only a fraction of lo-level labels compared to flat DAgger (Figure 6.2, right). Hierarchically Guided IL / RL. We evaluate hg-DAgger/Q with deep double 𝑄learning (DDQN, Van Hasselt et al., 2016) and prioritized experience replay (Schaul et al., 2015b) as the underlying RL procedure. Each subpolicy learner receives a pseudo-reward of 1 for each successful execution, corresponding to stepping through the correct door (e.g., door to the north if the subgoal is north) and negative reward for stepping into lava or through other doors. 6.1 (right) shows the learning progression of hg-DAgger/Q, implying two main observations. First, the number of hi-level labels rapidly increases initially and then flattens out after the learner becomes more successful, thanks to the availability of Inspectfull operation. As the hybrid algorithm makes progress and the learning agent passes the Inspectfull operation increasingly often, the algorithm starts saving significantly on expert feedback. Second, the number of hi-level labels is higher than for both hg-DAgger and h-BC. Inspectfull returns Fail often, especially during the early parts of training. This is primarily due to the slower learning speed of 𝑄-learning at the lo level, requiring more expert feedback at the hi level. This means that the hybrid algorithm is suited for settings where lo-level expert labels are either not available or more expensive than the hi-level labels. This is exactly the setting we analyze in the next section. In Appendix E.2, we compare hg-DAgger/Q with hierarchical RL (h-DQN, Kulkarni et al., 2016), concluding that h-DQN, even with significantly more lo-level samples, fails to reach success rate comparable to hg-DAgger/Q. Flat 𝑄-learning also fails in this setting, due to a long planning horizon and sparse rewards (Mnih et al., 2015). Hierarchically Guided IL / RL vs Hierarchical RL: Comparison on Montezuma’s Revenge Task Overview. Montezuma’s Revenge is among the most difficult Atari games for existing deep RL algorithms, and is a natural candidate for hierarchical approach 100 Learning Progression (random trial) 400 external rewards LO-level samples success rate 100% 80% 60% 40% 20% 0% 1000K 800K 600K 400K 200K 0K 0K Subgoal 1 Subgoal 2 (key) Subgoal 3 Subgoal 4 (door) 1K 2K 3K 4K 5K 6K 7K episode (HI-level labeling cost) 8K 9K 300 hg-DAgger/Q versus h-DQN (100 trials) hg-DAgger/Q 3rd quartile hg-DAgger/Q median h-DQN 200 100 0 0.0M 0.5M 1.0M 1.5M 2.0M 2.5M 3.0M 3.5M 4.0M LO-level reinforcement learning samples Figure 6.3: Montezuma’s revenge: hg-DAgger/Q versus h-DQN. (Left) Screenshot of Montezuma’s Revenge in black-and-white with color-coded subgoals. (Middle) Learning progression of hg-DAgger/Q in solving the first room of Montezuma’s Revenge for a typical successful trial. Subgoal colors match the left pane; success rate is the fraction of times the lo-level RL learner achieves its subgoal over the previous 100 attempts. (Right) Learning performance of hg-DAgger/Q versus hDQN (median and inter-quartile range). due to the sequential order of subtasks. Figure 6.3 (left) displays the environment and an annotated sequence of subgoals. The four designated subgoals are: go to bottom of the right stair, get the key, reverse path to go back to the right stair, then go to open the door (while avoiding obstacles throughout). The agent is given a pseudo-reward of 1 for each subgoal completion and -1 upon loss of life. We enforce that the agent can only have a single life per episode, preventing the agent from taking a shortcut after collecting the key (by taking its own life and re-initializing with a new life at the starting position, effectively collapsing the task horizon). Note that for this setting, the actual game environment is equipped with two positive external rewards corresponding to picking up the key (subgoal 2, reward of 100) and using the key to open the door (subgoal 4, reward of 300). Optimal execution of this sequence of subgoals requires more than 200 primitive actions. Unsurprisingly, flat RL algorithms often achieve a score of 0 on this domain (Mnih et al., 2015, 2016; Wang et al., 2016). hg-DAgger/Q versus h-DQN. Similar to the maze domain, we use DDQN with prioritized experience replay at the lo level of hg-DAgger/Q. We compare its performance with h-DQN using the same neural network architecture as (Kulkarni et al., 2016). Figure 6.3 (middle) shows the learning progression of our hybrid algorithm. The hi-level horizon 𝐻hi “ 4, so meta-controller is learnt from fairly few samples. Each episode roughly corresponds to one Labelhi query. Subpolicies are learnt in the order of subgoal execution as prescribed by the expert. We introduce a simple modification to 𝑄-learning on the lo level to speed up learning: the accumulation of experience replay buffer does not begin until the first 101 time the agent encounters positive pseudo-reward. During this period, in effect, only the meta-controller is being trained. This modification ensures the reinforcement learner encounters at least some positive pseudo-rewards, which boosts learning in the long horizon settings and should naturally work with any off-policy learning scheme (DQN, DDQN, Dueling-DQN). For a fair comparison, we introduce the same modification to the h-DQN learner (otherwise, h-DQN failed to achieve any reward). To mitigate the instability of DQN (see, for example, learning progression of subgoal 2 and 4 in Figure 6.3, middle), we introduce one additional modification. We terminate training of subpolicies when the success rate exceeds 90%, at which point the subgoal is considered learned. Subgoal success rate is defined as the percentage of successful subgoal completions over the previous 100 attempts. Figure 6.3 (right) shows the median and the inter-quartile range over 100 runs of hgDAgger/Q and hg-DQN.7 The lo-level sample sizes are not directly comparable with the middle panel, which displays the learning progression for a random successful run, rather than an aggregate over multiple runs. In all of our experiments, the performance of the imitation learning component is stable across many different trials, whereas the performance of the reinforcement learning component varies substantially. Subgoal 4 (door) is the most difficult to learn due to its long horizon whereas subgoals 1–3 are mastered very quickly, especially compared to h-DQN. Our algorithm benefits from hierarchical guidance and accumulates experience for each subgoal only within the relevant part of the state space, where the subgoal is part of an optimal trajectory. In contrast, h-DQN may pick bad subgoals and the resulting lo-level samples then “corrupt” the subgoal experience replay buffers and substantially slow down convergence.8 The number of hi-level labels in Figure 6.3 (middle) can be further reduced by using a more efficient RL procedure than DDQN at the lo level. In the specific example of Montezuma’s Revenge, the actual human effort is in fact much smaller, since the human expert needs to provide a sequence of subgoals only once (together with simple subgoal detectors), and then hi-level labeling can be done automatically. The human expert only needs to understand the high level semantics, and does not need to be able to play the game. 7 In Appendix E.2, we present additional plots, including 10 best runs of each algorithm, subgoal completion rate over 100 trials, and versions of Figure 6.3 (middle) for additional random instances. 8 In fact, we further reduced the number of subgoals of h-DQN to only two initial subgoals, but the agent still largely failed to learn even the second subgoal (see the appendix for details). 102 6.7 Discussion We have presented hierarchical guidance framework and shown how it can be used to speed up learning and reduce the cost of expert feedback in hierarchical imitation learning and hybrid imitation–reinforcement learning. Our approach can be extended in several ways. For instance, one can consider weaker feedback such as preference or gradient-style feedback (Fürnkranz et al., 2012; Loftin et al., 2016; Christiano et al., 2017), or a weaker form of imitation feedback, only saying whether the agent action is correct or incorrect, corresponding to bandit variant of imitation learning (Ross et al., 2011b). Our hybrid IL / RL approach relied on the availability of a subgoal termination predicate indicating when the subgoal is achieved. While in many settings such a termination predicate is relatively easy to specify, in other settings this predicate needs to be learned. We leave the question of learning the termination predicate, while learning to act from reinforcement feedback, open for future research. 103 Chapter 7 POLICY LEARNING WITH LATENT MODEL Summary. We study the problem of imitation learning from demonstrations of multiple coordinating agents. One key challenge in this setting is that learning a good model of coordination can be difficult, since coordination is often implicit in the demonstrations and must be inferred as a latent variable. We propose a joint approach that simultaneously learns a latent coordination model along with the individual policies. In particular, our method integrates unsupervised structure learning with conventional imitation learning. We illustrate the power of our approach on a difficult problem of learning multiple policies for fine-grained behavior modeling in team sports, where different players occupy different roles in the coordinated team strategy. We show that having a coordination model to infer the roles of players yields substantially improved imitation loss compared to conventional baselines. 7.1 Motivating Applications for Latent Structure The areas of multi-agent planning and control have witnessed a recent wave of strong interest due to the practical desire to deal with complex real-world problems, such as smart-grid control, autonomous vehicles planning, managing teams of robots for emergency response, among others. From the learning perspective, (cooperative) multi-agent learning is not a new area of research (Stone and Veloso, 2000; Panait and Luke, 2005). However, compared to the progress in conventional supervised learning and single-agent reinforcement learning, the successes of multiagent learning have remained relatively modest. Most notably, multi-agent learning suffers from extremely high dimensionality of both the state and actions spaces, as well as relative lack of data sources and experimental testbeds. The growing availability of data sources for coordinated multi-agent behavior, such as sports tracking data (Bialkowski et al., 2014), now enables the possibility of learning multi-agent policies from demonstrations, also known as multi-agent imitation learning. One particularly interesting aspect of domains such as team sports is that the agents must coordinate. For example, in the professional soccer setting depicted in Figure 7.1, different players must coordinate to assume different roles (e.g., defend left field). However, the roles and role assignment mechanism are unobserved from the demonstrations. Furthermore, the role for a player may change 104 Figure 7.1: Our motivating example of learning coordinating behavior policies for team sports from tracking data. Red is the attacking team, blue is the defending team, and yellow is the ball. during the same play sequence. In the control community, this issue is known as “index-free” multi-agent control (Kingston and Egerstedt, 2010). Motivated by these challenges, we study the problem of imitation learning for multiple coordinating agents from demonstrations. Many realistic multi-agent settings require coordination among collaborative agents to achieve some common goal (Guestrin et al., 2002; Kok et al., 2003). Beyond team sports, other examples include learning policies for game AI, controlling teams of multiple robots, or modeling collective animal behavior. As discussed above, we are interested in settings where agents have access to the outcome of actions from other agents, but the coordination mechanism is neither clearly defined nor observed, which makes the full state only partially observable. We propose a semi-supervised learning framework that integrates and builds upon conventional imitation learning and unsupervised, or latent, structure learning. The latent structure model encodes a coordination mechanism, which approximates the implicit coordination in the demonstration data. In order to make learning tractable, we develop an alternating optimization method that enables integrated and efficient training of both individual policies and the latent structure model. For learning individual policies, we extend reduction-based single-agent imitation learning approaches into multi-agent domain, utilizing powerful black-box supervised techniques such as deep learning as base routines. For latent structure learning, we develop a stochastic variational inference approach. We demonstrate the effectiveness of our method in two settings. The first is a synthetic experiment based on the popular predator-prey game. The second is a challenging task of learning multiple policies for team defense in professional 105 soccer, using a large training set1 of play sequences illustrated by Figure 7.1. We show that learning a good latent structure to encode implicit coordination yields significantly superior imitation performance compared to conventional baselines. To the best of our knowledge, this is the first time an imitation learning approach has been applied to jointly learn cooperative multi-agent policies at large scale. 7.2 Policy Learning Problem Formulation In coordinated multi-agent imitation learning, we have 𝐾 agents acting in coordination to achieve a common goal (or sequence of goals). Training data D consists of multiple demonstrations of 𝐾 agents. Importantly, we assume the identity (or indexing) of the 𝐾 experts may change from one demonstration to another. Each (unstructured) set of demonstrations is denoted by 𝑈 “ t𝑈1 , . . . , 𝑈𝐾 u, where 𝑈 𝑘 “ t𝑢 𝑡,𝑘 u𝑇𝑡“1 is the sequence of actions by agent 𝑘 at time 𝑡. Note that each set of demonstrations can have varying sequence length T. Let 𝐶 “ t𝑐 𝑡 u𝑇𝑡“1 be the context associated with each demonstration sequence. Policy Learning. Our ultimate goal is to learn a (largely) decentralized policy, but for clarity we first present the problem of learning a fully centralized multi-agent policy. Following the notation of (Ross et al., 2011a), let 𝜋p® ® 𝑠q :“ 𝑎® denote the joint policy that maps the joint state, 𝑠® “ r𝑠1 , . . . , 𝑠 𝐾 s, of all 𝐾 agents into 𝐾 actions 𝑎® “ r𝑎 1 , . . . , 𝑎 𝐾 s. The goal is to minimize imitation loss: L𝑖𝑚𝑖𝑡𝑎𝑡𝑖𝑜𝑛 “ E𝑠®„𝑑 𝜋® rℓp𝜋p® ® 𝑠qqs , where 𝑑 𝜋® denotes the distribution of states experienced by joint policy 𝜋® and ℓ is the imitation loss defined over the demonstrations (e.g., squared loss for deterministic policies, or cross entropy for stochastic policies). The decentralized setting decomposes the joint policy 𝜋® “ r𝜋1 , . . . , 𝜋 𝐾 s into 𝐾 policies, each tailored to a specific agent index or “role”.2 The loss function is then: L𝑖𝑚𝑖𝑡𝑎𝑡𝑖𝑜𝑛 “ 𝐾 ÿ E𝑠„𝑑 𝜋𝑘 rℓp𝜋 𝑘 p𝑠 𝑘 qqs . 𝑘 “1 Black-Box Policy Classes. In order to leverage powerful black-box policy classes such as random forests and deep learning, we take a learning reduction approach to 1 Data at http://www.stats.com/data-science/ and see video result at http:// hoangminhle.github.io 2 It is straightforward to extend our formulation to settings where multiple agents can occupy the same role, and where not all roles are occupied across all execution sequences. 106 training 𝜋. ® One consequence is that the state space representation 𝑠 “ r𝑠1 , . . . , 𝑠 𝐾 s must be consistently indexed, e.g., agent 𝑘 in one instance must correspond to agent 𝑘 in another instance. This requirement applies for both centralized and decentralized policy learning, and is often implicitly assumed in prior work on multi-agent learning. A highly related issue arises in distributed control of index-free coordinating robots, e.g., to maintain a defined formation (Kloder and Hutchinson, 2006; Kingston and Egerstedt, 2010). Motivating example: Soccer Domain. Consider the task of imitating professional soccer players, where training data includes play sequences from different teams and games. Context 𝐶 corresponds to the behavior of the opposing team and the ball. The data includes multiple sequences of 𝐾-set of trajectories 𝑈 “ t𝑈1 , 𝑈2 , . . . , 𝑈𝐾 u, where the actual identity of player generating 𝑈 𝑘 may change from one demonstration to the next. One important challenge for black-box policy learning is constructing an indexing mechanism over the agents to yield a consistent state representation. For example, the same index should correspond to the “left defender” in all instances. Otherwise, the inputs to the policy will be inconsistent, making learning difficult if not impossible. Note that barring extensive annotations or some heuristic rule-based definitions, it is unnatural to quantitatively define what makes a player “left defender”. In addition, even if we had a way to define who the “left defender” is, he may not stay in the same role during the same sequence. Role-based Indexing. We address index-free policy learning via role learning and role-based index assignment. To motivate our notion of role, let’s first consider the simplest indexing mechanism: one could equate role to agent identity. However, the data often comes from various sequences, with heterogeneous identities and teams of agents. Thus instead of learning identity-specific policies, it is more natural and data-efficient to learn a policy per role. However, a key challenge in learning policies directly is that the roles are undefined, unobserved, and could change dynamically within the same sequence. We thus view learning the coordination, via role assignment, as an unsupervised structured prediction problem. Coordination via Structured Role Assignment. Instead of handcrafting the definition of roles, we learn the roles in an unsupervised fashion, without attaching any semantic labels to the roles. At the same time, role transition should obey certain structural regularity, due to coordination. This motivates using graphical models to represent the coordination structure. 107 Figure 7.2: Alternating stochastic optimization training scheme for our semisupervised structure regularization model. Coordinated Policy Learning. We formulate the indexing mechanism as an assignment function A which maps the unstructured set 𝑈 and some probabilistic structured model 𝑞 to an indexed set of trajectory 𝐴 rearranged from 𝑈, i.e., A : t𝑈1 , .., 𝑈𝐾 u ˆ 𝑞 ÞÑ r𝐴1 , .., 𝐴𝐾 s , where the set t𝐴1 , .., 𝐴𝐾 u ” t𝑈1 , .., 𝑈𝐾 u. We view 𝑞 as a latent variable model that infers the role assignments for each set of demonstrations. Thus, 𝑞 drives the indexing mechanism A so that state vectors can be consistently constructed to facilitate optimizing for the imitation loss. We employ entropy regularization, augmenting the imitation loss with some low entropy penalty (Grandvalet et al., 2004; Dudik et al., 2004), yielding our overall objective: 𝐾 ÿ min E𝑠 𝑘 „𝑑 𝜋𝑘 rℓp𝜋 𝑘 p𝑠 𝑘 qq|A, Ds ´ 𝜆𝐻pA|Dq (7.1) 𝜋1 ,..,𝜋 𝐾 ,A 𝑘 “1 where both imitation loss and entropy are measured with respect to the state distribution induced by the policies, and D is training data. This objective can also be seen as maximizing the mutual information between latent structure and observed trajectories (Krause et al., 2010). 7.3 Policy Learning Algorithm with Structure Learning and Inference Optimizing (7.1) is challenging for two reasons. First, beyond the challenges inherited from single-agent settings, multi-agent imitation learning must account for multiple simultaneously learning agents, which is known to cause non-stationarity for multi-agent reinforcement learning (Busoniu et al., 2008). Second, the latent role assignment model, which forms the basis for coordination, depends on the actions of the learning policies, which in turn depend on the structured role assignment. 108 Algorithm 11 Coordinated Multi-Agent Imitation Learning Input: Multiple unstructured trajectory sets 𝑈 “ t𝑈1 , . . . , 𝑈𝐾 u with 𝑈 𝑘 “ t𝑢 𝑡,𝑘 u𝑇𝑡“1 and context 𝐶 “ t𝑐 𝑡 u𝑇𝑡“1 . Input: Graphical model 𝑞 with global/local parameters 𝜃 and 𝑧. Input: Initialized policies 𝜋 𝑘 , 𝑘 “ 1, . . . , 𝐾 Input: Step size sequence 𝜌𝑛 , 𝑛 “ 1, 2, . . . 1: repeat 2: r𝐴1 , . . . , 𝐴𝐾 s Ð Assignt𝑈1 , . . . , 𝑈𝐾 |𝑞p𝜃, 𝑧qu 3: r𝜋1 , . . . , 𝜋 𝐾 s Ð Learn r𝐴1 , . . . , 𝐴𝐾 , 𝐶s p1 , . . . , 𝐴 p𝐾 4: Roll-out 𝜋1 , . . . , 𝜋 𝐾 to obtain 𝐴 p 5: 𝐴 𝑘 Ð 𝐴 𝑘 @𝑘 p𝑘 with prob 𝜂 for 𝜂 Ñ 1) (Alternatively: 𝐴 𝑘 Ð 𝐴 6: 𝑞p𝜃, 𝑧q Ð LearnStructuret𝐴1 , . . . , 𝐴𝐾 , 𝐶, 𝜃, 𝜌𝑛 u 7: until No improvement on validation set output 𝐾 policies 𝜋1 , 𝜋2 , . . . , 𝜋 𝐾 We propose an alternating optimization approach to solving (7.1), summarized in Figure 7.2. The main idea is to integrate imitation learning with unsupervised structure learning by taking turns to (i) optimize for imitation policies while fixing a structured model (minimizing imitation loss), and (ii) re-train the latent structure model and reassign roles while fixing the learning policies (maximizing role assignment entropy). The alternating nature allows us to circumvent the circular dependency between policy learning and latent structure learning. Furthermore, for (i) we develop a stable multi-agent learning reduction approach. Approach Outline Algorithm 11 outlines our framework. We assume the latent structure model for computing role assignments is formulated as a graphical model. The multi-agent policy training procedure Learn utilizes a reduction approach, and can leverage powerful off-the-shelf supervised learning tools such as deep neural networks (Hochreiter and Schmidhuber, 1997). The structure learning LearnStructure and role assignment Assign components are based on graphical model training and inference. For efficient training, we employ alternating stochastic optimization (Hoffman et al., 2013; Johnson and Willsky, 2014; Beal, 2003) on the same mini-batches. Note that batch training can be deployed similarly, as illustrated by one of our experiments. We interleave the three components described above into a complete learning algorithm. Given an initially unstructured set of training data, an initialized set of policies, and prior parameters of the structure model, Algorithm 11 performs alternating structure optimization on each mini-batch (size 1 in Algorithm 11). 109 • Line 2: Role assignment is performed on trajectories t𝐴1 , . . . , 𝐴𝐾 u by running inference procedure (Algorithm 14). The result is an ordered set r𝐴1 , . . . , 𝐴𝐾 s, where trajectory 𝐴 𝑘 corresponds to policy 𝜋 𝑘 . • Line 3-5: Each policy 𝜋 𝑘 is updated using joint multi-agent training on the ordered set r𝐴1 , . . . , 𝐴𝐾 , 𝐶s (Algorithm 12). The updated models are executed to yield a rolled-out set of trajectories, which replace the previous set of trajectories t𝐴 𝑘 u. • Line 6: Parameters of latent structured model are updated from the rolled-out trajectories (Algorithm 13). The algorithm optionally includes a mixing step on line 5, where the rolled-out trajectories may replace the training trajectories with increasing probability approaching 1, which is similar to scheduled sampling (Bengio et al., 2015), and may help stabilize learning in the early phase of the algorithm. In our main experiment, we do not notice a performance gain using this option. Joint Multi-Agent Imitation Learning In this section we describe the Learn procedure for multi-agent imitation learning in Line 3 of Algorithm 11. As background, for single agent imitation learning, reduction-based methods operate by iteratively collecting a new data set D𝑛 at each round 𝑛 of training, consisting of state-action pairs p𝑠𝑡 , 𝑎 𝑡˚ q where 𝑎 𝑡˚ is some optimal or demonstrated action given state 𝑠𝑡 . A new policy can be formed by (i) combining a new policy from this data set D𝑛 with previously learned policy 𝜋 (Daumé III et al., 2009) or (ii) learning a new policy 𝜋 directly from the data set formed by aggregating D1 , . . . , D𝑛 (Ross et al., 2011a). Other variants exist although we do not discuss them here. The intuition behind the iterative reduction approach is to prevent a mismatch in training and prediction distributions due to sequential cascading errors (also called covariate-shift). The main idea is to use the learned policy’s own predictions in the construction of subsequent states, thus simulating the test-time performance during training. This mechanism enables the agent to learn a policy that is robust to its own mistakes. Reduction-based methods also accommodate any black-box supervised training subroutine. We focus on using expressive function classes such as Long 110 Algorithm 12 Joint Multi-Agent Imitation Learning: Learnp𝐴1 , 𝐴2 , . . . , 𝐴𝐾 , 𝐶q Input: Ordered actions 𝐴 𝑘 “ t𝑎 𝑡,𝑘 u𝑇𝑡“1 @𝑘, context t𝑐 𝑡 u𝑇𝑡“1 Input: Initialized policies 𝜋1 , . . . , 𝜋 𝐾 Input: base routine Trainp𝑆, 𝐴q mapping state to actions 1: Set increasing prediction horizon 𝑗 P t1, . . . , 𝑇u 2: for 𝑡 “ 0, 𝑗, 2 𝑗, . . . , 𝑇 do 3: for 𝑖 “ 0, 1, . . . , 𝑗 ´ 1 do 4: Roll-out 𝑎ˆ 𝑡 `𝑖,𝑘 “ 𝜋 𝑘 p𝑠ˆ𝑡 `𝑖´1,𝑘 q @ agent 𝑘 5: Cross-update for each policy 𝑘 P t1, . . . , 𝐾u 𝑠ˆ𝑡 `𝑖,𝑘 “ 𝜑 𝑘 pr𝑎ˆ 𝑡 `𝑖,1 , . . . , 𝑎ˆ 𝑡 `𝑖,𝑘 , . . . , 𝑎ˆ 𝑡 `𝑖,𝐾 , 𝑐 𝑡 `𝑖 sq 6: end for 7: Policy update for all agent 𝑘 𝑗 𝜋 𝑘 Ð Trainpt𝑠ˆ𝑡 `𝑖,𝑘 , 𝑎 𝑡˚`𝑖`1,𝑘 u𝑖“0 q 8: end for output 𝐾 updated policies 𝜋1 , 𝜋2 , . . . , 𝜋 𝐾 Short-Term Memory networks (LSTM) (Hochreiter and Schmidhuber, 1997) as the policy class.3 Algorithm 12 outlines the Learn procedure for stable multi-agent imitation learning. Assume we are given consistently indexed demonstrations 𝐴 “ r𝐴1 , . . . , 𝐴𝐾 s, where each 𝐴 𝑘 “ t𝑎 𝑡,𝑘 u𝑇𝑡“1 corresponds action of policy 𝜋 𝑘 . Let the corresponding expert ˚ . To lighten the notation, we denote the per-agent state vector by action be 𝑎 𝑡,𝑘 𝑠𝑡,𝑘 “ 𝜑 𝑘 pr𝑎 𝑡,1 , . . . , 𝑎 𝑡,𝑘 , . . . , 𝑎 𝑡,𝐾 , 𝑐 𝑡 sq4 Algorithm 12 employs a roll-out horizon 𝑗, which divides the entire trajectory into 𝑇{ 𝑗 segments. The following happens for every segment: • Iteratively perform roll-out at each time step 𝑖 for all 𝐾 policies (line 4) to obtain actions tp 𝑎𝑖,𝑘 u. • Each policy simultaneously updates its state p 𝑠𝑖,𝑘 , using the prediction from all other policies (line 5). • At the end of the current segment, all policies are updated using the error ˚ , for signal from the deviation between predicted 𝑎p𝑖,𝑘 versus expert action 𝑎𝑖,𝑘 all 𝑖 along the sub-segment (line 7). 3 Note that conventional training of LSTMs does not address the cascading error problem. While LSTMs are very good at sequence-to-sequence prediction tasks, they cannot naturally deal with the drifting of input state distribution drift caused by action output feedback in dynamical systems (Bengio et al., 2015). 4 Generally, state vector 𝑠 𝑡 ,𝑘 of policy 𝜋 𝑘 at time 𝑡 can be constructed as 𝑠 𝑡 ,𝑘 “ r𝜙 𝑘 pr𝑎 1:𝑡 ,1 , 𝑐 1:𝑡 sq, . . . , 𝜙 𝑘 pr𝑎 1:𝑡 ,𝐾 , 𝑐 1:𝑡 sqs 111 After policy updates, the training moves on to the next 𝑗-length sub-segment, using the freshly updated policies for subsequent roll-outs. The iteration proceeds until the end of the sequence is reached. In the outer loop the roll-out horizon 𝑗 is incremented. Two key insights behind our approach are: • In addition to the training-prediction mismatch issue in single-agent learning, each agent’s prediction must also be robust to imperfect predictions from other agents. This non-stationarity issue also arises in multi-agent reinforcement learning (Busoniu et al., 2008) when agents learn simultaneously. We perform joint training by cross-updating each agent’s state using previous predictions from other agents. • Many single-agent imitation learning algorithms assume the presence of a dynamic oracle to provide one-step corrections 𝑎 𝑡˚ along the roll-out trajectories. In practice, dynamic oracle feedback is very expensive to obtain and some recent work have attempted to relax this requirement (Le et al., 2016a; Ho and Ermon, 2016). Without dynamic oracles, the rolled-out trajectory can deviate significantly from demonstrated trajectories when the prediction horizon 𝑗 is large (« 𝑇), leading to training instability. Thus 𝑗 is gradually increased to allow for slowly learning to make good sequential predictions over longer horizons. For efficient training, we focus on stochastic optimization, which can invoke base routine Train multiple times and thus naturally accommodates varying 𝑗. Note that the batch-training alternatives to Algorithm 12 can also employ similar training schemes, with similar theoretical guarantees lifted to the multi-agent case. The Appendix shows how to use DAgger (Ross et al., 2011a) for Algorithm 12, which we used for our synthetic experiment. Coordination Structure Learning The coordination mechanism is based on a latent structured model that governs the role assignment. The training and inference procedures seek to address two main issues: • LearnStructure: unsupervised learning a probabilistic role assignment model 𝑞. 112 • Assign: how 𝑞 informs the indexing mechanism so that unstructured trajectories can be mapped to structured trajectories amenable to Algorithm 12. Given an arbitrarily ordered set of trajectories 𝑈 “ t𝑈1 , . . . , 𝑈𝐾 , 𝐶u, let the coordination mechanism underlying each such 𝑈 be governed by a true unknown model 𝑝, with global parameters 𝜃. We suppress the agent/policy subscript and consider a generic featurized trajectory 𝑥𝑡 “ r𝑢 𝑡 , 𝑐 𝑡 s @𝑡. Let the latent role sequence for the same agent be 𝑧 “ 𝑧1:𝑇 . At any time 𝑡, each agent is acting according to a latent role ¯ which are the local parameters to the structured 𝑧𝑡 „ Categoricalt1̄, 2̄, . . . , 𝐾u, model. Ideally, role and index asignment can be obtained by calculating the true posterior 𝑝p𝑧|𝑥, 𝜃q, which is often intractable. We instead aim to approximate 𝑝p𝑧|𝑥, 𝜃q by a simpler distribution 𝑞 via techniques from stochastic variational inference (Hoffman et al., 2013), which allows for efficient stochastic training on mini-batches that can naturally integrate with our imitation learning subroutine. In variational inference, posterior approximation is often cast as optimizing over a simpler model class Q, via searching for parameters 𝜃 and 𝑧 that maximize the evidence lower bound (ELBO) L: L p𝑞p𝑧, 𝜃qq fi E𝑞 rln 𝑝p𝑧, 𝜃, 𝑥qs ´ E𝑞 rln 𝑞p𝑧, 𝜃qs ď ln 𝑝p𝑥q Maximizing L is equivalent to finding 𝑞 P Q to minimize the KL divergence KL p𝑞p𝑧, 𝜃|𝑥q||𝑝p𝑧, 𝜃|𝑥qq. We focus on the structured mean-field variational family, which factorizes 𝑞 as 𝑞p𝑧, 𝜃q “ 𝑞p𝑧q𝑞p𝜃q. This factorization breaks the dependency between 𝜃 and 𝑧, but not between single latent states 𝑧𝑡 , unlike variational inference for i.i.d data (Kingma and Welling, 2013). Training to learn model parameters The procedure to learn the parameter of our structured model is summarized in Algorithm 13. Parameter learning proceeds via alternating updates over the factors 𝑞p𝜃q and 𝑞p𝑧q, while keeping other factor fixed. Stochastic variational inference performs such updates efficiently in mini-batches. We slightly abuse notations and overload 𝜃 for the natural parameters of global parameter 𝜃 in the exponential family. Assuming the usual conjugacy in the exponential family, the stochastic natural gradient takes a convenient form (line 2 of Algo 13, and derivation in Appendix), where 𝑡p𝑧, 𝑥q is the vector of sufficient statistics, 𝑏 is a vector of scaling 113 Algorithm 13 Structure Learning: LearnStructure t𝑈1 , . . . , 𝑈𝐾 , 𝐶, 𝜃, 𝜌u ÞÑ 𝑞p𝜃, 𝑧q Input: 𝑋 𝑘 “ t𝑥𝑡,𝑘 u𝑇𝑡“1 “ tr𝑢 𝑡,𝑘 , 𝑐 𝑡 su @𝑡, 𝑘.𝑋 “ t𝑋 𝑘 u𝐾𝑘“1 Graphical model parameters 𝜃, stepsize 𝜌 1: Local update: compute 𝑞p𝑧q via message-passing while fixing 𝜃 (See Appendix for derivations) 2: Global parameter update: via natural gradient ascent 𝜃 Ð 𝜃p1 ´ 𝜌q ` 𝜌p𝜃 𝑝𝑟𝑖𝑜𝑟 ` 𝑏 J E𝑞p𝑧q r𝑡p𝑧, 𝑥qsq output Updated model 𝑞p𝜃, 𝑧q “ 𝑞p𝜃q𝑞p𝑧q factors adjusting for the relative size of the mini-batches. Here the global update assumes optimal local update 𝑞p𝑧q has been computed. Fixing the global parameters, the local updates are based on message-passing over the underlying graphical model. The exact mathematical derivation depends on the specific graph structure. The simplest scenario is to assume independence among 𝑧𝑡 ’s, which resembles naive Bayes. In our experiments, we instead focus on Hidden Markov Models to capture first-order dependencies in role transitions over play sequences. In that case, line 1 of Algorithm 13 resembles running the forwardbackward algorithm to compute the update 𝑞p𝑧q. The forward-backward algorithm in the local update step takes 𝑂p𝐾 2𝑇q time for a chain of length 𝑇 and 𝐾 hidden states. For completeness, derivation of parameter learning for HMMs is included in the Appendix. Inference for role and index assignment We can compute two types of inference on a learned 𝑞: ¯ 𝑇, Role inference. Compute the most likely role sequence t𝑧𝑡,𝑘 u𝑇𝑡“1 P t1̄, . . . , 𝐾u e.g., using Viterbi (or dynamic programming-based forward message passing for graph structures). This most likely role sequence for agent 𝑘, which is the lowdimensional representation of the coordination mechanism, can be used to augment the contextual feature t𝑐 𝑡 u𝑇𝑡“1 for each agent’s policy training. Role-based Index Assignment Transform the unstructured set 𝑈 into an ordered set of trajectories 𝐴 to facilitate the imitation learning step. This is the more important task for the overall approach. The intuitive goal of an indexing mechanism is to facilitate consistent agent trajectory to policy mapping. Assume for notational convenience that we want index 𝑘 assigned to an unique agent who is most likely ¯ Our inference technique rests on the well-known Linear Assignassuming role 𝑘. ment Problem (Papadimitriou and Steiglitz, 1982), which is solved optimally via the 114 Algorithm 14 Multi-Agent Role Assignment: Assign t𝑈1 , . . . , 𝑈𝐾 |𝑞u ÞÑ r𝐴1 , . . . , 𝐴𝐾 s Input: Approximate inference model 𝑞. Unordered trajectories 𝑈 “ t𝑈 𝑘 u𝐾𝑘“1 . 1: Calculate cost matrix 𝑀 P R𝐾 ˆ𝐾 per equation 7.2 2: A Ð MinCostAssignmentp𝑀q output 𝐴 𝑘 “ 𝑈A p 𝑘 q @𝑘 “ 1, 2, . . . , 𝐾 Kuhn-Munkres algorithm. Specifically, construct the cost matrix 𝑀 as: 𝑀 “ 𝑀1 d 𝑀2 (7.2) « ff 𝑇 ź “ ‰ ¯ “ ¯ 𝑞p𝑥𝑡,𝑘 |𝑧𝑡,𝑘 “ 𝑘q 𝑀1 “ 𝑞pt𝑥𝑡,𝑘 u|𝑧𝑡,𝑘 “ 𝑘q 𝑡 “1 « ff 𝑇 ÿ ‰ ¯ “ ¯ 𝑀2 “ ln 𝑞pt𝑥𝑡,𝑘 u|𝑧𝑡,𝑘 “ 𝑘q ln 𝑞p𝑥𝑡,𝑘 |𝑧𝑡,𝑘 “ 𝑘q “ 𝑡 “1 ¯ d is the Hadamard product, and matrices where 𝑘 “ 1, . . . , 𝐾, 𝑘¯ “ 1̄, . . . , 𝐾, 𝑀1 , 𝑀2 take advantage of the Markov property of the graphical model. Now solving the linear assignment problem for cost matrix 𝑀, we obtain the matching A from role 𝑘¯ to index 𝑘, such that the total cost per agent is minimized. From here, we rearrange the unordered set t𝑈1 , . . . , 𝑈𝐾 u to the ordered sequence r𝐴1 , . . . , 𝐴𝐾 s ” r𝑈A p1q , . . . , 𝑈A p𝐾 q s according to the minimum cost mapping. To see why this index assignment procedure results in an increased entropy in the original objective (7.1), notice that: 𝐾 ÿ 𝐻pA|Dq « ´ ¯ ¯ ¯ 𝑃p 𝑘q𝑞pAp𝐴 𝑘 q “ 𝑘q log 𝑞pAp𝐴 𝑘 q “ 𝑘q 𝑘¯ “1 𝐾 1 ÿ ¯ 𝑘q, “´ 𝑀p 𝑘, 𝐾 ¯ 𝑘 “1 where we assume each latent role 𝑘¯ has equal probability. The RHS increases from the linear assignment and consequent role assignment procedure. Our inference procedure to perform role assignment is summarized in Algorithm 14. 7.4 Experiments We present empirical results from two settings. The first is a synthetic setting based on predator-prey, where the goal is to imitate a coordinating team of predators. The second is a large-scale imitation learning setting from player trajectores in professional soccer games, where the goal is to imitate defensive team play. 115 Predator-Prey Domain Setting. The predator-prey problem, also frequently called the Pursuit Domain (Benda, 1985), is a popular setting for multi-agent reinforcement learning. The traditional setup is with four predators and one prey, positioned on a grid board. At each time step, each agent has five moves: N,S,E,W or no move. The world is toroidal: the agents can move off one end of the board and come back on the other end. Agents make move simultaneously, but two agents cannot occupy the same position, and collisions are avoided by assigning a random move priority to the agents at each time step. The predators can capture the prey only if the prey is surrounded by all four Figure 7.3: predators. The goal of the predators is to capture the prey as fast as possible, which necessarily requires coordination. Data. The demonstration data is collected from 1000 game instances, where four experts, indexed 1 to 4, are prescribed the consistent and coordinated role as illustrated in the capture state of Figure 7.3. In other words, agent 1 would attempt to capture the prey on the right hand side, which allows for one fixed role for each expert throughout the game. However, the particular role assignment is hidden from the imitation learning task. Each expert is then exhaustively trained using Value Iteration (Sutton and Barto, 1998) in the reinforcement learning setting, with the reward of 1 if the agent is in the position next to the prey according to its defined role, and 0 otherwise. A separate set of 100 games was collected for evaluation. A game is terminated after 50 time steps if the predators fail to capture the prey. In the test set, the experts fail to capture the prey in 2% of the games, and on average take 18.3 steps to capture the prey. Experiment Setup. For this experiment, we use the batch version of Algorithm 11 (see appendix) to learn to imitate the experts using only demonstrations. Each policy is represented by a random forest of 20 trees, and were trained over 10 iterations. The expert correction for each rolled-out state is collected via Value Iteration. The experts thus act as dynamic oracles, which result in a multi-agent training setting analogous to DAgger (Ross et al., 2011a). We compare two versions of multi-agent imitation learning: • Coordinated Training. We use our algorithm, with the latent structure model represented by a discrete Hidden Markov Model with binomial emission. We 116 Figure 7.4: Comparing performance in Predator-Prey between our approach and unstructured multi-agent imitation learning, as a function of the number of training rounds. Our approach demonstrates both significantly lower failure rates as well as lower average time to success (for successful trials). use Algorithm 14 to maximize the role consistency of the dynamic oracles across different games. • Unstructured Training. An arbitrary role is assigned to each dynamic oracle for each game, i.e., the agent index is meaningless. In both versions, training was done using the same data aggregation scheme and batch training was conducted using the same random forests configuration. Results. Figure 7.4 compares the test performance of our method versus unstructured multi-agent imitation learning. Our method quickly approaches expert performance (average 22 steps with 8% failure rate in the last iteration), whereas unstructured multi-agent imitation learning performance did not improve beyond the first iteration (average 42 steps with 70% failure rate). Note that we even gave the unstructured baseline some advantage over our method, by forcing the prey to select the moves last after all predators make decisions (effectively making the prey slower). Without this advantage, the unstructured policies fail to capture the prey almost 100% of the time. Also, if the same restriction is applied to the policies obtained from our method, performance would be on par with the experts (100% success rate, with similar number of steps taken). Multi-agent Imitation Learning for Soccer Setting. Soccer is a popular domain for multi-agent learning. RoboCup, the robotic and simulation soccer platform, is perhaps the most popular testbed for multi-agent 117 Figure 7.5: Experimental results on soccer domain. We see that using coordination substantially improves the imitation loss, and that the decentralized policy is comparable to the centralized. reinforcement learning research to date (Stone, 2016). The success of MARL has been limited, however, due to the extremely high dimensionality of the problem. In this experiment, we aim to learn multi-agent policies for team soccer defense, based on tracking data from real-life professional soccer (Bialkowski et al., 2014). Data. We use the tracking data from 45 games of real professional soccer from a recent European league. The data was chunked into sequences with one team attacking and the other defending. Our goal is to learn up to 10 policies for team defense (11 players per team, minus the goal keeper). The training data consists of 7500 sets of trajectories 𝐴 “ t𝐴1 , . . . , 𝐴10 u , where 𝐴 𝑘 “ t𝑎 𝑡,𝑘 u𝑇𝑡“1 is the sequence of positions of one defensive player, and 𝐶 is the context consisting of opponents and the ball. Overall, there are about 1.3 million frames at 10 frames per second. The average sequence length is 176 steps, and the maximum is 1480. Experiment Setup. Each policy is represented by a recurrent neural network structure (LSTM), with two hidden layers of 512 units each. As LSTMs generally require fixed-length input sequences, we further chunk each trajectory into subsequences of length 50, with overlapping window of 25 time steps. The joint multi-agent imitation learning procedure follows Algorithm 12 closely. In this setup, without access to dynamic oracles for imitation learning in the style of SEARN (Daumé III et al., 2009) and DAgger (Ross et al., 2011a), we gradually increase the horizon of the rolled-out trajectories from 1 to 10 steps look-ahead. Empirically, this has the effect of stabilizing the policy networks early in training, and limits the cascading errors caused by rolling-out to longer horizons. The structured model component is learned via stochastic variational inference on a continuous HMM, where the per-state emission distribution is a mixture of 118 Figure 7.6: Result of 10 coordinated imitation policies, corresponding with Figure 7.1. White is the rolled-out imitation policies. Gaussians. Training and inference operate on the same mini-batches used for joint policy learning. We compare against two variations. The first employs centralized policy that aggregates the state vectors of all decentralized learner and produces the actions for all players, i.e., a multi-task policy. The centralized approach generally requires more model parameters, but is potentially much more accurate. The second variation is to not employ joint multi-agent training: we modify Algorithm 12 to not cross-update states between agents, and each role is trained conditioned on the ground truth of the other agents. Results. Figure 7.5 shows the results. Our coordinated learning approach substantially outperforms conventional imitation learning without structured coordination. The imitation loss measures average distance of roll-outs and ground truth in meters (note the typical size of soccer field is 110 ˆ 70 meters). As expected, average loss increases with longer sequences, due to cascading errors. However, this error scales sub-linearly with the length of the horizon, even though the policies were trained on sequences of length 50. Note also that the performance difference between decentralized and centralized policies is insignificant compared to the gap between coordinated and unstructured policies, further highlighting the benefits of structured coordination in multi-agent settings. The loss of a single network, non-joint training scheme is very large and thus omitted from Figure 7.5 (see the appendix). Visualizations. Imitation loss, of course, is not a full reflection of the quality of the learned policies. Unlike predator-prey, the long-term reward signal is not available, so we rely on visual inspection as part of evaluation. Figure 7.6 overlays policy prediction on top of the actual game sequence from Figure 7.1. Additional 119 Figure 7.7: Components of role distributions, corresponding to a popular formation arrangement in professional soccer test examples are included in supplemental videos 5. We note that learned policies are qualitatively similar to the ground truth demonstrations, and can be useful for applications such as counterfactual analysis (Le et al., 2017). Figure 7.7 displays the Gaussian components of the underlying HMM. The components correspond to the dominant modes of the roles assigned. Unlike the predator-prey domain, roles can be switched during a sequence of play. See the appendix for more details on role swap frequency. 7.5 Related Work in Multi-Agent Learning Context The problem of multi-agent imitation learning has not been widely considered, perhaps with the exception of (Chernova and Veloso, 2007) which focused on very different applications and technical challenges (i.e., learning a model of a joint task by collecting samples from direct interaction with teleoperating human teachers). The actual learning algorithm there requires the learner to collect enough data points from human teachers for confident classification of task. It is not clear how well the proposed method would translate to other domains. Index-free policy learning is generally difficult for black-box machine learning techniques. Some recent work has called attention to the importance of order to learning when input or output are sets (Vinyals et al., 2015), motivated by classic algorithmic and geometric problems such as learning to sort a set of numbers, or finding convex hull for a set of points, where no clear indexing mechanism exists. Other permutation invariant approaches include those for standard classification (Shivaswamy and Jebara, 2006). 5 Additional videos can be seen at http://hoangle.info 120 7.6 Limitations and Discussions In principle, the training and inference of the latent structure model can accommodate different types of graphical models. However, the exact procedure varies depending on the graph structure. It would be interesting to find domains that can benefit from more general graphical models. Another possible direction is to develop fully end-to-end differentiable training methods that can accommodate our index-free policy learning formulation, especially deep learning-based method that could provide computational speed-up compared to traditional graphical model inference. One potential issue with the end-to-end approach is the need to depart from a learning-reductions style approach. Although we addressed learning from demonstrations in this chapter, the proposed framework can also be employed for generative modeling, or more efficient structured exploration for reinforcement learning. Along that line, our proposed method could serve as a useful component of general reinforcement learning, especially in multi-agent settings where traditional exploration-based approaches such as Qlearning prove computationally intractable. 121 Chapter 8 CONCLUDING REMARKS In this dissertation, we have developed three groups of techniques for structured policy learning. To summarize, in Chapter 2 and 3, we discussed value-based structured policy learning, which imposes constraints on overall performance of the policy as an external specification. In Chapter 4 and 5, we showed that building structural constraints directly into the policy class can yield both certifiable policy performance, as well as improved data efficiency in the setting of constrained imitation learning. In Chapter 6 and 7, we demonstrated that exploiting partial knowledge of the model can significantly improve the data efficiency of learning, as well as alleviating the generally difficult imperfect observation scenarios. As an overarching message, policy learning has much to gain from bringing systematic structure into learning. Not only can we achieve more data-efficient learning, but the learning agent can also better capture what we ultimately care about in reallife applications. Through our applications, structured policy learning has shown encouraging performance, and is an overall more reasonable approach compared to the alternative paradigm of end-to-end learning, which has been powered by the recent advances in deep learning. Looking forward, we do have several interesting questions that arise from the approach of this thesis: • How do we reason about generalization power of policy learning when we incorporate structural constraints? How do we develop a further unifying perspectives to bring together different types of structural constraints presented in this work? An interesting research direction as a proof of concept would be bringing value-based and policy-based approaches together, or alternatively, indentifying methods to unify different model-based structure into one common framework for better analysis and algorithm design • In addition to methodological advances, there is perhaps no better replacement for real-world policy learning than trying to solve real-world problems. The research community would benefit from expanding into better benchmarking testbeds that reflect realistic constraints. More broadly, this opens up the 122 question of how do we interface policy learning methods with established methods from other disciplines. One of the emerging lessons from our recent work is that we have much to learn from the algorithmic techniques that other disciplines have come up with for decision making, such as game theory, or program synthesis, or control theory and others. Such a fusion of knowledge can potentially accelerate the advent of a learning-based systems of the future that best serve the common interests 123 BIBLIOGRAPHY Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004. Pieter Abbeel and Andrew Y Ng. Exploration and apprenticeship learning in reinforcement learning. In International Conference on Machine Learning (ICML), 2005. Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22–31, 2017. Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, pages 1638–1646, 2014. Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, 2018. Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In ThirtySecond AAAI Conference on Artificial Intelligence, 2018. Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999. Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo MK Martin, Mukund Raghothaman, Sanjit A Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. In 2013 Formal Methods in Computer-Aided Design, pages 1–8. IEEE, 2013. Rajeev Alur, Arjun Radhakrishna, and Abhishek Udupa. Scaling enumerative program synthesis via divide and conquer. In Tools and Algorithms for the Construction and Analysis of Systems - 23rd International Conference, TACAS 2017, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2017, Uppsala, Sweden, April 22-29, 2017, Proceedings, Part I, pages 319–336, 2017. Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In ICML, 2017. OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020. 124 Kiam Heong Ang, Gregory Chong, and Yun Li. Pid control system analysis, design, and technology. IEEE transactions on control systems technology, 13(4):559–576, 2005. András Antos, Csaba Szepesvári, and Rémi Munos. Fitted q-iteration in continuous action-space mdps. In Advances in neural information processing systems, pages 9–16, 2008a. András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1):89–129, 2008b. Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5): 469–483, 2009. Karl Johan Åström and Tore Hägglund. Automatic tuning of simple regulators with specifications on phase and amplitude margins. Automatica, 20(5):645–651, 1984. J Andrew Bagnell, Sham M Kakade, Jeff G Schneider, and Andrew Y Ng. Policy search by dynamic programming. In Neural Information Processing Systems (NIPS), 2003. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 2426, 2017, Conference Track Proceedings, 2017. Heejung Bang and James M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962–973, 2005. doi: 10.1111/j.15410420.2005.00377.x. Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension bounds for piecewise linear neural networks. In Proceedings of the 22nd Annual Conference on Learning Theory (COLT 2017), 2017. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems, pages 2494–2504, 2018. Heinz H Bauschke, Patrick L Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011. Matthew James Beal. Variational algorithms for approximate Bayesian inference. University of London United Kingdom, 2003. Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003. 125 Richard Bellman, Irving Glicksberg, and Oliver Gross. On the “bang-bang” control problem. Quarterly of Applied Mathematics, 14(1):11–18, 1956. Miroslav Benda. On optimal cooperation of knowledge sources. Technical Report BCS-G2010-28, 1985. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171–1179, 2015. Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems, pages 908–918, 2017. Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA, 2005. Alina Bialkowski, Patrick Lucey, Peter Carr, Yisong Yue, and Iain Matthews. “win at home and draw away”: Automatic formation analysis highlighting the differences in home and away team behaviors. In MIT Sloan Sports Analytics Conference (SSAC), 2014. Lars Blackmore, Masahiro Ono, and Brian C Williams. Chance-constrained optimal path planning with obstacles. IEEE Transactions on Robotics, 27(6):1080–1094, 2011. David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112 (518):859–877, 2017. Roderick Bloem, Krishnendu Chatterjee, Thomas A. Henzinger, and Barbara Jobstmann. Better quality in synthesis through quantitative objectives. In Computer Aided Verification, 21st International Conference, CAV 2009, Grenoble, France, June 26 - July 2, 2009. Proceedings, pages 140–156, 2009. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016. Léon Bottou, Jonas Peters, Joaquin Qui nonero Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research (JMLR), 14:3207–3260, 2013. Philippe Bougerol and Nico Picard. Strict stationarity of generalized autoregressive processes. The Annals of Probability, pages 1714–1730, 1992. 126 Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems Man and Cybernetics Part C Applications and Reviews, 38(2):156, 2008. Rich Caruana and Alexandru Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In International Conference on Machine Learning (ICML), 2006. Rich Caruana, Nikos Karampatziakis, and Ainur Yessenalina. An empirical evaluation of supervised learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pages 96–103, 2008. Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume III, and John Langford. Learning to search better than your teacher. In ICML, 2015. Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pages 2249–2257, 2011. Swarat Chaudhuri, Martin Clochard, and Armando Solar-Lezama. Bridging boolean and quantitative synthesis using smoothed proof search. In POPL, pages 207–220, 2014. Jianhui Chen and Peter Carr. Mimicking human camera operators. In IEEE Winter Conference Applications of Computer Vision (WACV), 2015. Jianhui Chen, Hoang M. Le, Peter Carr, Yisong Yue, and James J. Little. Learning online smooth predictors for real-time camera planning using recurrent decision trees. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning (ICML), 2019. C Cheng, X Yan, N Wagener, and B Boots. Fast policy learning through imitation and reinforcement. In Uncertainty in artificial intelligence, 2019a. Ching-An Cheng, Xinyan Yan, Nathan Ratliff, and Byron Boots. Predictor-corrector policy optimization. In International Conference on Machine Learning, pages 1151–1161, 2019b. Ching-An Cheng, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Accelerating imitation learning with predictive models. In Conference on Artificial Intelligence and Statistics (AISTATS), 2019c. 127 Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. In International Conference on Machine Learning, pages 1141–1150, 2019d. Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. In International Conference on Machine Learning, pages 1141–1150, 2019e. Sonia Chernova and Manuela Veloso. Multiagent collaborative task learning through imitation. In Proceedings of the fourth International Symposium on Imitation in Animals and Artifacts, pages 74–79, 2007. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017. Antonio Criminisi, Jamie Shotton, and Ender Konukoglu. Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning. Foundations and Trends in Computer Graphics and Vision, 7(2–3):81–227, 2012. Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018. Hal Daumé III, John Langford, and Daniel Marcu. Search-based structured prediction. Machine learning, 75(3):297–325, 2009. Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In NIPS, 1993. Leonardo Mendonça de Moura and Nikolaj Bjørner. Z3: An Efficient SMT Solver. In TACAS, pages 337–340, 2008. Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 179–186. Omnipress, 2012. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In Proceedings of the 34th International Conference on Machine LearningVolume 70, pages 990–998. JMLR. org, 2017. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019. 128 Thomas G Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Intell. Res.(JAIR), 13(1):227–303, 2000. Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665–1674, 2019. Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. Inversecsg: automatic conversion of 3d models to CSG trees. ACM Trans. Graph., 37(6):213:1–213:16, 2018. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329–1338, 2016. Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pages 1087–1098, 2017. John C Duchi, Shai Shalev-Shwartz, Yoram Singer, and Ambuj Tewari. Composite objective mirror descent. In COLT, pages 14–26, 2010. Miroslav Dudik, Steven J Phillips, and Robert E Schapire. Performance guarantees for regularized maximum entropy density estimation. In International Conference on Computational Learning Theory, pages 472–486. Springer, 2004. Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In International Conference on Machine Learning (ICML), 2011a. Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 1097–1104. Omnipress, 2011b. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 207–219, 2017. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. In Advances in Neural Information Processing Systems, pages 6059–6068, 2018. Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503–556, 2005. 129 Amir M Farahmand, Mohammad Ghavamzadeh, Shie Mannor, and Csaba Szepesvári. Regularized policy iteration. In Advances in Neural Information Processing Systems, pages 441–448, 2009. Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. arXiv preprint arXiv:1802.03493, 2018a. Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning (ICML), 2018b. John K. Feser, Swarat Chaudhuri, and Isil Dillig. Synthesizing data structure transformations from input-output examples. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015, pages 229–239, 2015. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49–58, 2016. Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, pages 385–394. Society for Industrial and Applied Mathematics, 2005. Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29:79–103, 1999. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. Springer, 2001. Ronan Fruit and Alessandro Lazaric. Exploration–exploitation in mdps with options. arXiv preprint arXiv:1703.08667, 2017. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052–2062, 2019. Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89(1-2):123–156, 2012. Vamsidhar Reddy Gaddam, Ragnhild Eg, Ragnar Langseth, Carsten Griwodz, and Pål Halvorsen. The cameraman operating my virtual camera is artificial: Can the machine be as good as a human&quest. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 11(4):56, 2015. Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015. 130 Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. In NIPS, volume 17, pages 529–536, 2004. Carlos Guestrin, Michail Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In ICML, volume 2, pages 227–234, 2002. Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program synthesis. Foundations and Trends in Programming Languages, 4(1-2):1–119, 2017. Zhaohan Guo, Philip S Thomas, and Emma Brunskill. Using options and covariance testing for long horizon off-policy policy evaluation. In Advances in Neural Information Processing Systems, pages 2492–2501, 2017. László Györfi, Michael Kohler, Adam Krzyzak, and Harro Walk. A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006. David Ha and Jürgen Schmidhuber. arXiv:1803.10122, 2018. World models. arXiv preprint Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages 1352–1361, 2017. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in neural information processing systems, pages 6765–6774, 2017. John Michael Hammersley and David Christopher Handscomb. Monte carlo methods. 1964. Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning (ICML), 2019. Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. Q(lambda) with off-policy corrections. In International Conference on Algorithmic Learning Theory (ALT), 2016. Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. In ICLR, 2016. David Haussler. Sphere packing numbers for subsets of the boolean n-cube with bounded vapnik-chervonenkis dimension. Journal of Combinatorial Theory, Series A, 69(2):217–232, 1995. He He, Jason Eisner, and Hal Daume. Imitation learning by coaching. In Neural Information Processing Systems (NIPS), 2012. Ruijie He, Emma Brunskill, and Nicholas Roy. Puma: Planning under uncertainty with macro-actions. In AAAI, 2010. 131 Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.02705, 2019. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565–4573, 2016. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. Matt Hoffman and David Blei. Structured stochastic variational inference. CoRR abs/1404.4114, 2014. Matthew D Hoffman, David M Blei, Chong Wang, and John William Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(1): 1303–1347, 2013. Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663–685, 1952. Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. In Neural Information Processing Systems (NIPS), 2013. Susmit Jha, Sumit Gulwani, Sanjit A Seshia, and Ashish Tiwari. Oracle-guided component-based program synthesis. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pages 215–224. ACM, 2010. Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652–661, 2016a. Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning (ICML), 2016b. Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In Proceedings of the 34th International Conference on Machine LearningVolume 70, pages 1704–1713. JMLR. org, 2017. 132 Matthew James Johnson and Alan S Willsky. Stochastic variational inference for bayesian time series models. In ICML, pages 1854–1862, 2014. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning (ICML), 2002. Sham Machandranath Kakade et al. On the sample complexity of reinforcement learning. PhD thesis, University of London London, England, 2003. Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. arXiv preprint arXiv:1908.08526, 2019a. Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon: Double reinforcement learning in infinite-horizon processes. arXiv preprint arXiv:1909.05850, 2019b. Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), 2018. Joseph DY Kang, Joseph L Schafer, et al. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical science, 22(4):523–539, 2007. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. Peter Kingston and Magnus Egerstedt. Index-free multi-agent systems: An eulerian approach. IFAC Proceedings Volumes, 43(19):215–220, 2010. Jyrki Kivinen and Manfred K Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1):1–63, 1997. Stephen Kloder and Seth Hutchinson. Path planning for permutation-invariant multirobot formations. IEEE Transactions on Robotics, 22(4):650–665, 2006. Sven Koenig and Reid G Simmons. The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms. Machine Learning, 22(1-3):227–250, 1996. Jelle R Kok, Matthijs TJ Spaan, Nikos Vlassis, et al. Multi-robot decision making using coordination graphs. In Proceedings of the 11th International Conference on Advanced Robotics, ICAR, volume 3, pages 1124–1129, 2003. Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008–1014, 2000. 133 Andreas Krause, Pietro Perona, and Ryan G Gomes. Discriminative clustering by regularized information maximization. In Advances in neural information processing systems, pages 775–783, 2010. Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. In Advances in Neural Information Processing Systems, pages 1840–1848, 2016. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NIPS, pages 3675–3683, 2016. Michail Lagoudakis and Ronald Parr. Reinforcement learning as classification: Leveraging modern classifiers. In International Conference on Machine Learning (ICML), 2003a. Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of machine learning research, 4(Dec):1107–1149, 2003b. Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45–73. Springer, 2012. John Langford and Bianca Zadrozny. Relating reinforcement learning performance to classification performance. In International Conference on Machine Learning (ICML), 2005. Alessandro Lazaric and Marcello Restelli. Transfer from multiple mdps. In Advances in Neural Information Processing Systems, pages 1746–1754, 2011. Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of lstd. In ICML-27th International Conference on Machine Learning, pages 615–622, 2010. Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of least-squares policy iteration. Journal of Machine Learning Research, 13(Oct):3041–3074, 2012. Hoang Le, Andrew Kang, Yisong Yue, and Peter Carr. Smooth imitation learning for online sequence prediction. In Proceedings of The 33rd International Conference on Machine Learning, pages 680–688, 2016a. Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703–3712, 2019a. 134 Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703–3712, 2019b. Hoang M. Le, Andrew Kang, Yisong Yue, and Peter Carr. Smooth imitation learning for online sequence prediction. In International Conference on Machine Learning (ICML), 2016b. Hoang M. Le, Peter Carr, Yisong Yue, and Patrick Lucey. Data-driven ghosting using deep imitation learning. In MIT Sloan Sports Analytics Conference (SSAC), 2017. Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995. Wee Sun Lee, Peter L Bartlett, and Robert C Williamson. Efficient agnostic learning of neural networks with bounded fan-in. IEEE Transactions on Information Theory, 42(6):2118–2132, 1996. Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pages 1071–1079, 2014. Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In International Conference on Web Search and Data Mining (WSDM), 2011. Lihong Li, Remi Munos, and Csaba Szepesvari. Toward minimax off-policy value estimation. In Artificial Intelligence and Statistics, pages 608–616, 2015. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. Lydia Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pages 3156–3164, 2018a. Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Neural Information Processing Systems (NIPS), 2018b. 135 Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems, pages 5361–5371, 2018c. Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arXiv:1904.08473, 2019. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114–4124, 2019. Robert Loftin, Bei Peng, James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning. In AAMAS, 2016. Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H Chi. Off-policy learning in two-stage recommender systems. In Proceedings of The Web Conference 2020, pages 463–473, 2020. Sridhar Mahadevan and Bo Liu. Sparse q-learning with mirror descent. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, pages 564–573. AUAI Press, 2012. Odalric-Ambrym Maillard, Rémi Munos, Alessandro Lazaric, and Mohammad Ghavamzadeh. Finite-sample analysis of bellman residual minimization. In Proceedings of 2nd Asian Conference on Machine Learning, pages 299–314, 2010. Llew Mason, Jonathan Baxter, Peter L Bartlett, and Marcus Frean. Functional gradient techniques for combining hypotheses. In Neural Information Processing Systems (NIPS), 1999. Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. arXiv preprint arXiv:1911.05815, 2019. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, pages 1928–1937, 2016. 136 Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2012. William H Montgomery and Sergey Levine. Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems, pages 4008–4016, 2016. Rémi Munos. Error bounds for approximate policy iteration. In ICML, volume 3, pages 560–567, 2003. Rémi Munos. Performance bounds in l_p-norm for approximate value iteration. SIAM journal on control and optimization, 46(2):541–561, 2007. Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(May):815–857, 2008. Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054–1062, 2016. Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. Neural sketch learning for conditional program generation. In ICLR, 2018. Susan A Murphy, Mark J van der Laan, James M Robins, and Conduct Problems Prevention Research Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1410–1423, 2001. Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pages 2315–2325, 2019. Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In ICRA, 2017. Arkadii Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley, 1983. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278–287, 1999. Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-treat policies. arXiv preprint arXiv:1905.09751, 2019. Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881–4890, 2019. 137 Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, 2018. Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, and J Balaram. Chanceconstrained dynamic programming with application to risk-aware robotic space exploration. Autonomous Robots, 39(4):555–571, 2015. Dirk Ormoneit and Śaunak Sen. Kernel-based reinforcement learning. Machine learning, 49(2-3):161–178, 2002. Cosmin Paduraru. Off-policy evaluation in Markov decision processes. PhD thesis, McGill University Libraries, 2013. Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntak Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. In Robotics: science and systems, 2018. Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 11(3):387–434, 2005. Christos H Papadimitriou and Kenneth Steiglitz. Combinatorial optimization: algorithms and complexity. Courier Corporation, 1982. Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research, 12(3):441–450, 1987. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. arXiv preprint arXiv:1611.01855, 2016. Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231–2240, 2017. Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Simto-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018. Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé FrezzaBuet. Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7 (3):7, 2011. Oleksandr Polozov and Sumit Gulwani. Flashmeta: a framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015, pages 107–126, 2015. 138 Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305–313, 1989. Michael JD Powell and J Swann. Weighted uniform sampling—a monte carlo technique for reducing variance. IMA Journal of Applied Mathematics, 2(3): 228–236, 1966. Niranjani Prasad, Li-Fang Cheng, Corey Chivers, Michael Draugelis, and Barbara E Engelhardt. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. arXiv preprint arXiv:1704.06300, 2017. Doina Precup, Richard S Sutton, and Satinder P Singh. Eligibility traces for offpolicy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759–766. Morgan Kaufmann Publishers Inc., 2000. Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 417–424. Morgan Kaufmann Publishers Inc., 2001. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pages 5099–5108, 2017. Nathan Ratliff, David Silver, and J. Andrew Bagnell. Learning to search: Functional gradient techniques for imitation learning. Autonomous Robots, 27(1):25–53, 2009. Veselin Raychev, Pavol Bielik, Martin T. Vechev, and Andreas Krause. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016, pages 761–774, 2016. Martin Riedmiller. Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning, pages 317–328. Springer, 2005. Martin Riedmiller, Thomas Gabel, Roland Hafner, and Sascha Lange. Reinforcement learning for robot soccer. Autonomous Robots, 27(1):55–73, 2009. Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67–113, 2013. 139 Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Conference on Artificial Intelligence and Statistics (AISTATS), 2010. Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. Stephane Ross, Geoff Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Conference on Artificial Intelligence and Statistics (AISTATS), 2011a. Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, pages 627–635, 2011b. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pages 1312–1320, 2015a. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015b. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889–1897, 2015. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107–194, 2012. Jack Sherman and Winifred J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. Ann. Math. Statist., 21 (1):124–127, 03 1950. doi: 10.1214/aoms/1177729893. Pannagadatta K Shivaswamy and Tony Jebara. Permutation invariant svms. In Proceedings of the 23rd international conference on Machine learning, pages 817–824. ACM, 2006. Xujie Si, Yuan Yang, Hanjun Dai, Mayur Naik, and Le Song. Learning a metasolver for syntax-guided program synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016. 140 David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676): 354, 2017. Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. Combinatorial sketching for finite programs. In ASPLOS, pages 404–415, 2006. Jonathan Sorg, Richard L Lewis, and Satinder P Singh. Reward design via online gradient ascent. In Advances in Neural Information Processing Systems, pages 2190–2198, 2010. Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In Neural Information Processing Systems (NIPS), 2010. Peter Stone. What’s hot at RoboCup. In Association for the Advancement of Artificial Intelligence (AAAI), 2016. Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. Autonomous Robots, 8(3):345–383, 2000. Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. arXiv preprint arXiv:1703.01030, 2017. Wen Sun, J Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning. In International Conference on Learning Representations (ICLR), 2018a. Wen Sun, Geoffrey J Gordon, Byron Boots, and J Bagnell. Dual policy iteration. In Advances in Neural Information Processing Systems, pages 7059–7069, 2018b. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018a. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018b. Richard S Sutton, Doina Precup, and Satinder P Singh. Intra-option learning about temporally abstract actions. In ICML, volume 98, pages 556–564, 1998. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semimdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999. 141 Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057–1063, 2000. Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16(1):1731–1755, 2015. Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems, pages 3632–3642, 2017. Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In NIPS, pages 1449–1456, 2008. Umar Syed and Robert E Schapire. A reduction from apprenticeship learning to classification. In Neural Information Processing Systems (NIPS), 2010. Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews. A deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):93, 2017. Guy Tennenholtz, Shie Mannor, and Uri Shalit. Off-policy evaluation in partially observable environments. arXiv preprint arXiv:1909.03739, 2019. Philip Thomas. Reinforcement learning: An introduction, 2015. Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning (ICML), 2016a. Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139–2148, 2016b. Philip S Thomas, William C Dabney, Stephen Giguere, and Sridhar Mahadevan. Projected natural actor-critic. In Advances in neural information processing systems, pages 2337–2345, 2013. Philip S Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. 2017. Masatoshi Uehara and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. arXiv preprint arXiv:1910.12809, 2019. 142 Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. In Advances in Neural Information Processing Systems, pages 8687–8698, 2018. Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI, volume 2, page 5. Phoenix, AZ, 2016. Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5052–5061, 2018. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161, 2017. Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391, 2015. Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind Blog, 2019. John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior (commemorative edition). Princeton university press, 2007. Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudík. Optimal and adaptive offpolicy evaluation in contextual bandits. In International Conference on Machine Learning, pages 3589–3597, 2017. Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In ICML, pages 1995–2003, 2016. MA Wiering. Multi-agent reinforcement learning for traffic light control. In International Conference on Machine Learning (ICML), 2000. Herman Wold. A study in the analysis of stationary time series, 1939. Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. TORCS, The Open Racing Car Simulator. http://www.torcs.org, 2014. Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. In Neural Information Processing Systems (NIPS), 2019. 143 Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. arXiv preprint arXiv:2001.11113, 2020. Stephan Zheng, Yisong Yue, and Patrick Lucey. Generating long-term trajectories using deep hierarchical networks. In NIPS, 2016. He Zhu, Zikang Xiong, Stephen Magill, and Suresh Jagannathan. An inductive synthesis framework for verifiable reinforcement learning. In ACM Conference on Programming Language Design and Implementation (SIGPLAN), 2019. Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, CMU, 2010. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 928–936, 2003. 144 Appendix A APPENDIX TO CHAPTER 2 In this appendix section, we provide additional discussion and detailed proofs for the results presented in Chapter 2. A.1 Equivalence between Regularization and Constraint Satisfaction Formulating Different Regularized Policy Learning Problems as Constrained Policy Learning In this section, we provide connections between regularized policy learning and our constrained formulation (OPT). Although the main chapter in the thesis focuses on batch policy learning, here we are agnostic between online and batch learning settings. Entropy regularized RL. The standard reinforcement learning objective, either in ˚ that minimizes the long-term cost online or batch setting, is to find a policy 𝜋std (equivalent to maximizing the accumuted rewards): ÿ ˚ 𝜋std “ arg min Ep𝑥𝑡 ,𝑎𝑡 q„𝜋 r𝑐p𝑥𝑡 , 𝑎 𝑡 qs “ arg min Ep𝑥,𝑎q„ 𝜇 𝜋 r𝑐p𝑥, 𝑎qs 𝜋 𝑡 𝜋 Maximum entropy reinforcement learning (Haarnoja et al., 2017) augments the cost with an entropy term, such that the optimal policy maximizes its entropy ˚ at each visited state: 𝜋MaxEnt “ arg min𝜋 Ep𝑥,𝑎q„ 𝜇 𝜋 r𝑐p𝑥, 𝑎qs ´ 𝜆Hp𝜋p¨|𝑥qq. As discuseed by (Haarnoja et al., 2017), the goal is for the agent to maximize the entropy of the entire trajectory, and not greedily maximizing entropy at the current time step (i.e., Boltzmann exploration). Maximum entropy policy learning was first proposed by (Ziebart et al., 2008; Ziebart, 2010) in the context of learning from expert demonstrations. Entropy regulazed RL/IL is equivalent to our problem (OPT) by simply set 𝐶p𝜋q “ Ep𝑥𝑡 ,𝑎𝑡 q„𝜋 r𝑐p𝑥𝑡 , 𝑎 𝑡 qs (standard RL objective), and 𝑔p𝑥, 𝑎q “ 𝜋p𝑎|𝑥q log 𝜋p𝑎|𝑥q, thus 𝐺p𝜋q “ ´Hp𝜋q ď 𝜏 Smooth imitation learning (& Regularized imitation learning). This is a constrained imitation learning problem studied by (Le et al., 2016a): learning to mimic smooth behavior in continuous space from human desmonstrations. The data collected from human demonstrations is considered to be fixed and given a priori, thus the imitation learning task is also a batch policy learning problem. 145 The proposed approach from (Le et al., 2016a) is to view policy learning as a function regularization problem: policy 𝜋 “ p 𝑓 , 𝑔q is a combination of functions 𝑓 and ℎ, where 𝑓 belongs to some expressive function class F (e.g., decision trees, neural networks) and ℎ P H with certifiable smoothness property (e.g., linear models). Policy learning is the solution to the functional regularization problem 𝜋 “ arg min 𝑓 ,𝑔 E𝑥 „ 𝜇 𝜋 } 𝑓 p𝑥q ´ 𝜋 𝐸 p𝑥q} ` 𝜆 }ℎp𝑥q ´ 𝜋 𝐸 p𝑥q}, where 𝜋 𝐸 is the expert policy. This constrained imitation learning setting is equivalent to our problem (OPT) as follows: 𝐶p𝜋q “ 𝐶pp 𝑓 , ℎqq “ E𝑥 „ 𝜇 𝜋 } 𝑓 p𝑥q ´ 𝜋 𝐸 p𝑥q} and 𝐺p𝜋q “ 𝐺pp 𝑓 , ℎqq “ minℎ1 PH }ℎ1 p𝑥q ´ 𝜋 𝐸 p𝑥q} ď 𝜏 Regularizing RL with expert demonstrations / Learning from imperfect demonstrations. Efficient exploration in RL is a well-known challenge. Expert demonstrations provide a way around online exploration to reduce the sample complexity for learning. However, the label budget for expert demonstrations may be limited, resulting in a sparse coverage of the state space compared to what the online RL agent can explore (Hester et al., 2018). Additionally, expert demonstrations may be imperfect (Oh et al., 2018). Some recent work proposed to regularize standard RL objective with some deviation measure between the learning policy and (sparse) expert data (Hester et al., 2018; Oh et al., 2018; Henaff et al., 2019). For clarity we focus on the regularized RL objective for Q-learning in (Hester et al., 2018), which is defined as 𝐽p𝜋q “ 𝐽𝐷𝑄 p𝑄q`𝜆 1 𝐽𝑛 p𝑄q`𝜆 2 𝐽𝐸 p𝑄q`𝜆 3 𝐽 𝐿2 p𝑄q, where 𝐽𝐷𝑄 p𝑄q is the standard deep Q-learning loss, 𝐽𝑛 p𝑄q is the n-step return loss, 𝐽𝐸 p𝑄q is the imitation learning loss defined as 𝐽𝐸 p𝑄q “ max𝑎PA r𝑄p𝑥, 𝑎q ` ℓp𝑎 𝐸 , 𝑎q ´ 𝑄p𝑥, 𝑎 𝐸 qs, and 𝐽 𝐿2 p𝑄q is an L2 regularization loss applied to the Q-network to prevent overfitting to a small expert dataset. The regularization parameters 𝜆’s are obtained by hyperparameter tuning. This approach provides a bridge between RL and IL, whose objective functions are fundamentally different (see AggreVate by (Ross and Bagnell, 2014) for an alternative approach). We can cast this problem into (OPT) as: 𝐶p𝜋q “ 𝐶𝐷𝑄 p𝑄q ` 𝜆 3𝐶 𝐿2 p𝑄q (standard RL objective), and two constraints: 𝑔1 p𝜋q “ E𝑥 „ 𝜇 𝜋 rmax𝑎PA 𝑄p𝑥, 𝑎q ` ℓp𝑎 𝐸 , 𝑎q ´ 𝑄p𝑥, 𝑎 𝐸 qs, and 𝑔2 p𝑥, 𝑎q “ E𝑥 „ 𝜇 𝜋 r𝑐 𝑡 ` 𝛾𝑐 𝑡 `1 ` . . . ` 𝛾 𝑛´1 𝑐 𝑡 `𝑛´1 ` min1𝑎 𝛾 𝑛 𝑄p𝑥𝑡 `𝑛 , 𝑎 1 q ´ 𝑄p𝑥𝑡 , 𝑎qs Here 𝑔1 captures the loss w.r.t. expert demonstrations and 𝑔2 reflects the n-step return constraint. More generally, one can define the imitation learning constraint as 𝐺p𝜋q “ E𝑥 „ 𝜇 𝜋 ℓp𝜋p𝑥q, 𝜋 𝐸 p𝑥qq for an appropriate divergence definition between 𝜋p𝑥q 146 and 𝜋 𝐸 p𝑥q (defined at states where expert demonstrations are available). Conservative policy improvement. Many policy search algorithms perform small policy update steps, requiring the new policy 𝜋 to stay within a neighborhood of the most recent policy iterate 𝜋 𝑘 to ensure learning stability (Levine and Abbeel, 2014; Schulman et al., 2015; Montgomery and Levine, 2016; Achiam et al., 2017). This simply corresponds to the definition of 𝐺p𝜋q “ distancep𝜋, 𝜋 𝑘 q ď 𝜏, where distance is typically 𝐾 𝐿-divergence or total variation distance between the distribution induced by 𝜋 and 𝜋 𝑘 . For 𝐾 𝐿-divergence, the single timestep cost 𝜋 p𝑎 |𝑥 q 𝑔p𝑥, 𝑎q “ ´𝜋p𝑎|𝑥q logp 𝜋𝑘p𝑎|𝑥 q q Equivalence of Regularization and Constraint Viewpoint - Proof of Proposition 2.2.1 Regularization ùñ Constraint: Let 𝜆 ą 0 and 𝜋 ˚ be optimal policy in Regularization. Set 𝜏 “ 𝐺p𝜋 ˚ q. Suppose that 𝜋 ˚ is not optimal in Constraint. Then D𝜋 P Π such that 𝐺p𝜋q ď 𝜏 and 𝐶p𝜋q ă 𝐶p𝜋 ˚ q. We then have 𝐶p𝜋q ` 𝜆J 𝐺p𝜋q ă 𝐶p𝜋 ˚ q ` 𝜆J 𝜏 “ 𝐶p𝜋 ˚ q ` 𝜆J 𝐺p𝜋 ˚ q which contradicts the optimality of 𝜋 ˚ for Regularization problem. Thus 𝜋 ˚ is also the optimal solution of the Constraint problem. Constraint ùñ Regularization: Given 𝜏 and let 𝜋 ˚ be the corresponding optimal solution of the Constraint problem. The Lagrangian of Constraint is given by 𝐿p𝜋, 𝜆q “ 𝐶p𝜋q`𝜆J 𝐺p𝜋q, 𝜆 ě 0. We then have 𝜋 ˚ “ arg min max 𝐿p𝜋, 𝜆q. 𝜋 PΠ 𝜆 ě0 Let 𝜆˚ “ arg max min 𝐿p𝜋, 𝜆q 𝜆 ě0 𝜋 PΠ Slater’s condition implies strong duality. By strong duality and the strong max-min property (Boyd and Vandenberghe, 2004), we can exchange the order of maximization and minimization. Thus 𝜋 ˚ is the optimal solution of min 𝜋 PΠ 𝐶p𝜋q ` p𝜆˚ qJ p𝐺p𝜋q ´ 𝜏q Removing the constaint p𝜆˚ qJ 𝜏, we have that 𝜋 ˚ is the optimal solution of the Regularization problem with 𝜆 “ 𝜆˚ . And since 𝜋 ˚ ‰ arg min 𝐶p𝜋q, we must have 𝜆˚ ě 0. 𝜋 PΠ 147 A.2 Convergence Proofs Convergence of Meta-algorithm - Proof of Proposition 2.3.1 Let us evaluate the empirical primal-dual gap of the Lagrangian after 𝑇 iterations: 1ÿ 𝐿p𝜋𝑡 , 𝜆q 𝜆 𝑇 𝑡 𝑜p𝑇q 1ÿ ď 𝐿p𝜋𝑡 , 𝜆𝑡 q ` 𝑇 𝑡 𝑇 𝑜p𝑇q 1ÿ ď 𝐿p𝜋, 𝜆𝑡 q ` @𝜋 P Π 𝑇 𝑡 𝑇 (A.2) 𝑜p𝑇q “ 𝐿p𝜋, 𝜆p𝑇 q ` 𝑇 (A.4) max 𝐿p𝜋ˆ𝑇 , 𝜆q “ max 𝜆 @𝜋 (A.1) (A.3) Equations (A.1) and (A.4) are due to the definition of 𝜋ˆ𝑇 and 𝜆p𝑇 and linearity of 𝐿p𝜋, 𝜆q wrt 𝜆 and the distribution over policies in Π. Equation (A.2) is due to the no-regret property of Online-algorithm. Equation (A.3) is true since 𝜋𝑡 is best response wrt 𝜆𝑡 . Since equation (A.4) holds for all 𝜋, we can conclude that for 𝑇 𝑜p𝑇 q sufficiently large such that 𝑇 ď 𝜔, we have max𝜆 𝐿p𝜋ˆ𝑇 , 𝜆q ď min𝜋 𝐿p𝜋, 𝜆p𝑇 q ` 𝜔 , which will terminate the algorithm. Note that we always have max𝜆 𝐿p𝜋ˆ𝑇 , 𝜆q ě 𝐿p𝜋ˆ𝑇 , 𝜆p𝑇 q ě min𝜋 𝐿p𝜋, 𝜆p𝑇 q. Algorithm 1’s convergence rate depends on the regret bound of the Online-algorithm ? procedure. Multiple algorithms exist with regret scaling as Ωp 𝑇q (e.g., online gradient descent with regularizer, variants of online mirror descent). In that case, the algorithm will terminate after 𝑂p 𝜔12 q iterations. Empirical Convergence Analysis of Main Algorithm - Proof of Theorem 2.4.1 By choosing normalized exponentiated gradient as the online learning subroutine, we have the following regret bound after 𝑇 iterations of the main algorithm 2 (chapter 𝑚 `1 2 of (Shalev-Shwartz et al., 2012)) for any 𝜆 P R` , }𝜆}1 “ 𝐵: 𝑇 𝑇 1 ÿp 1 ÿp 𝐿p𝜋𝑡 , 𝜆q ď 𝐿p𝜋𝑡 , 𝜆𝑡 q ` 𝑇 𝑡 “1 𝑇 𝑡 “1 𝐵 logp𝑚 `1q s 2 𝐵𝑇 ` 𝜂𝐺 𝜂 𝑇 (A.5) 148 𝐵 logp𝑚`1q s2 `𝜂𝐺 𝐵𝑇 𝜂 p 𝜆q in Denote 𝜔𝑇 “ to simplify notations. By the linearity of 𝐿p𝜋, 𝑇 both 𝜋 and 𝜆, we have for any 𝜆 that 𝑇 𝑇 eqn p 𝐴.5q 1 ÿ linearity 1 ÿ p p p 𝑡 , 𝜆𝑡 q ` 𝜔𝑇 𝐿p𝜋ˆ𝑇 , 𝜆q “ 𝐿p𝜋𝑡 , 𝜆q ď 𝐿p𝜋 𝑇 𝑡 “1 𝑇 𝑡 “1 𝑇 best response 𝜋𝑡 ď 1 ÿp linearity p 𝜋ˆ𝑇 , 𝜆p𝑇 q ` 𝜔𝑇 𝐿p𝜋ˆ𝑇 , 𝜆𝑡 q ` 𝜔𝑇 “ 𝐿p 𝑇 𝑡 “1 p 𝜋ˆ𝑇 , 𝜆q ď 𝐿p p 𝜋ˆ𝑇 , 𝜆p𝑇 q ` 𝜔𝑇 . Since this is true for any 𝜆, max𝜆 𝐿p On the other hand, for any policy 𝜋, we also have 𝑇 𝑇 best response 𝜋𝑡 1 ÿ 1 ÿp p 𝑡 , 𝜆𝑡 q p 𝜆p𝑇 q linearity 𝐿p𝜋, 𝜆𝑡 q 𝐿p𝜋 ě 𝐿p𝜋, “ 𝑇 𝑡 “1 𝑇 𝑡 “1 𝑇 eqn p 𝐴.5q 1 ÿ ě 𝑇 𝑡 “1 p 𝑡 , 𝜆p𝑇 q ´ 𝜔𝑇 linearity p 𝜋ˆ𝑇 , 𝜆p𝑇 q ´ 𝜔𝑇 𝐿p𝜋 “ 𝐿p p 𝜆p𝑇 q ě 𝐿p p 𝜋ˆ𝑇 , 𝜆p𝑇 q ´ 𝜔𝑇 , leading to Thus min𝜋 𝐿p𝜋, p 𝜋ˆ𝑇 , 𝜆q ´ min 𝐿p𝜋, p 𝜆p𝑇 q ď 𝐿p p 𝜋ˆ𝑇 , 𝜆p𝑇 q ` 𝜔𝑇 ´ p 𝐿p p 𝜋ˆ𝑇 , 𝜆p𝑇 q ´ 𝜔𝑇 q “ 2𝜔𝑇 max 𝐿p 𝜋 𝜆 After 𝑇 iterations of the main algorithm 2, therefore, the empirical primal-dual gap is bounded by 2 p 𝜋ˆ𝑇 , 𝜆q ´ min 𝐿p𝜋, p 𝜆p𝑇 q ď max 𝐿p 𝜆 𝜋 𝐵 logp𝑚 `1q s 2 𝐵𝑇 ` 2𝜂𝐺 𝜂 𝑇 In particular, if we want the gap to fall below a desired threshold 𝜔, setting the online s2 logp𝑚 `1q 16𝐵2 𝐺 learning rate 𝜂 “ 4𝐺s𝜔2 𝐵 will ensure that the algorithm converges after 𝜔2 iterations. 149 A.3 End-to-end Generalization Analysis of Main Algorithm In this section, we prove the following full statement of theorem 2.4.4 of the main s to be the bound of chapter 2. Note that to lessen notation, we define 𝑉s “ 𝐶s ` 𝐵𝐺 value functions under considerations in algorithm 2. Theorem A.3.1 (Generalization bound of algorithm 2). Let 𝜋 ˚ be the optimal policy to problem OPT. Let 𝐾 be the number of iterations of FQE and FQI. Let 𝜋ˆ be the policy returned by our main algorithm 2, with termination threshold 𝜔. s2 𝐾 p𝑚 `1q ¨𝑉s4 p log 𝛿 For any 𝜖 ą 0, 𝛿 P p0, 1q, when 𝑛 ě 24¨214 ` dimF log 320𝜖 2𝑉 ` 𝜖2 logp14𝑒pdimF ` 1qqq, we have with probability at least 1 ´ 𝛿: 𝐶p𝜋q ˆ ď 𝐶p𝜋 ˚ q ` 𝜔 ` and p4 ` 𝐵q𝛾 a s p 𝐶 𝜌 𝜖 ` 2𝛾 𝐾 {2𝑉q 3 p1 ´ 𝛾q a 𝑉s ` 𝜔 𝛾 1{2 2𝛾 𝐾 {2𝑉s p 𝐶𝜌 𝜖 ` q 𝐺p𝜋q ˆ ď 𝜏`2 ` 𝐵 p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 ř Let 𝜋ˆ “ 𝑇1 𝑡 𝜋𝑡 be the returned policy 𝑇 iterations, with corresponding dual variable ř 𝜆p “ 𝑇1 𝑡 𝜆𝑡 . By the stopping condition, the empirical duality gap is less than some threshold 𝜔, p p p 𝜋, p 𝜆q p ď 𝜔 where 𝐿p𝜋, p 𝜆q “ 𝐶p𝜋q i.e., max 𝐿p ˆ 𝜆q ´ min 𝐿p𝜋, ` 𝜆J p𝐺p𝜋q ´ 𝜋 PΠ 𝑚`1 𝜆 P R` ,}𝜆}1 “ 𝐵 𝜏q. We first show that the returned policy approximately satisfies the constraints. The proof of theorem A.3.1 will make use of the following empirical constraint satisfaction bound: Lemma A.3.2 (Empirical constraint satisfactions). Assume that the constraints p 𝐺p𝜋q ď 𝜏 are feasible. Then the returned policy 𝜋ˆ approximately satisfies all constraints 𝐶s ` 𝜔 max pp 𝑔𝑖 p𝜋q ˆ ´ 𝜏𝑖 q ď 2 𝑖 “1:𝑚 `1 𝐵 Proof. We consider max pp 𝑔𝑖 p𝜋q ˆ ´ 𝜏𝑖 q ą 0 (otherwise the lemma statement is triv𝑖 “1:𝑚 `1 p 𝜋, p ially true). The termination condition implies that 𝐿p ˆ 𝜆q´ max 𝑚`1 𝜆PR` ,}𝜆}1 “ 𝐵 p 𝜋, 𝐿p ˆ 𝜆q ě ´𝜔 p 𝜋q ùñ 𝜆pJ p𝐺p ˆ ´ 𝜏pq ě max 𝑚`1 𝜆PR` ,}𝜆}1 “ 𝐵 p 𝜋q 𝜆J p𝐺p ˆ ´ 𝜏pq ´ 𝜔 (A.6) 150 𝑔𝑖 p𝜋q ˆ ´ 𝜏𝑖 s Relaxing the RHS of equation (A.6) by setting 𝜆r 𝑗s “ 𝐵 for 𝑗 “ arg max rp 𝑖 “1:𝑚 `1 and 𝜆r𝑖s “ 0 @𝑖 ‰ 𝑗 yields: p 𝜋q ˆ ´ 𝜏q 𝐵 max rp 𝑔𝑖 p𝜋q ˆ ´ 𝜏𝑖 s ´ 𝜔 ď 𝜆pJ p𝐺p 𝑖 “1:𝑚 `1 (A.7) p Given 𝜋 such that 𝐺p𝜋q ď 𝜏, also by the termination condition: p 𝜋, p ´ 𝐿p𝜋, p 𝜆q p ď 𝐿p ˆ 𝜆q max 𝑚`1 𝜆 P R` ,}𝜆}1 “ 𝐵 p 𝜋, p 𝜆q p ď𝜔 𝐿p ˆ 𝜆q ´ min 𝐿p𝜋, 𝜋 PΠ Thus implies p p p p 𝜋, p ď 𝐿p𝜋, p 𝜆q p ` 𝜔 “ 𝐶p𝜋q 𝐿p ˆ 𝜆q ` 𝜆pJ p𝐺p𝜋q ´ 𝜏q ď 𝐶p𝜋q `𝜔 (A.8) combining what we have from equation (A.8) and (A.7): p 𝜋q p 𝜋q p ` 𝜔 ´ 𝐶p p 𝜋q p 𝜋, p ´ 𝐶p 𝑔𝑖 p𝜋q ˆ ´ 𝜏p𝑖 s ´ 𝜔 ď 𝜆pJ p𝐺p ˆ ´ 𝜏pq “ 𝐿p ˆ 𝜆q ˆ ď 𝐶p𝜋q ˆ 𝐵 max rp 𝑖 “1:𝑚 `1 p p 𝜋q Rearranging and bounding 𝐶p𝜋q ď 𝐶s and 𝐶p ˆ ď ´𝐶s finishes the proof of the lemma. We now return to the proof of theorem A.3.1, our task is to lift empirical error to generalization bound for main objective and constraints. Denote by 𝜖 𝐹𝑄𝐸 the (generalization) error introduced by the Fitted Q Evaluation procedure (algorithm 3) and 𝜖 𝐹𝑄𝐼 the (generalization) error introduced by the Fitted Q Iteration procedure (algorithm 15). For now we keep 𝜖 𝐹𝑄𝐸 and 𝜖 𝐹𝑄𝐼 unspecified (to be specified shortly). That is, for each 𝑡 “ 1, 2, . . . , 𝑇, we have with probability at least 1 ´ 𝛿: ˚ J ˚ 𝐶p𝜋𝑡 q ` 𝜆J 𝑡 p𝐺p𝜋𝑡 q ´ 𝜏q ď 𝐶p𝜋 q ` 𝜆 𝑡 p𝐺p𝜋 q ´ 𝜏q ` 𝜖 𝐹𝑄𝐼 Since 𝜋 ˚ satisfies the constraints, i.e., 𝐺p𝜋 ˚ q ´ 𝜏 ď 0 componentwise, and 𝜆𝑡 ě 0, we also have with probability 1 ´ 𝛿 ˚ 𝐿p𝜋𝑡 , 𝜆𝑡 q “ 𝐶p𝜋𝑡 q ` 𝜆J 𝑡 p𝐺p𝜋𝑡 q ´ 𝜏q ď 𝐶p𝜋 q ` 𝜖 𝐹𝑄𝐼 (A.9) Similarly, with probability 1 ´ 𝛿, all of the following inequalities are true p 𝑡 q ` 𝜖 𝐹𝑄𝐸 ě 𝐶p𝜋𝑡 q ě 𝐶p𝜋 p 𝑡 q ´ 𝜖 𝐹𝑄𝐸 𝐶p𝜋 (A.10) p 𝑡 q ` 𝜖 𝐹𝑄𝐸 1 ě 𝐺p𝜋𝑡 q ě 𝐺p𝜋 p 𝑡 q ´ 𝜖 𝐹𝑄𝐸 1 (row wise for all 𝑚 constraintsq 𝐺p𝜋 (A.11) 151 Thus with probability at least 1 ´ 𝛿 J p J p 𝐿p𝜋𝑡 , 𝜆𝑡 q “ 𝐶p𝜋𝑡 q ` 𝜆J 𝑡 p𝐺p𝜋𝑡 q ´ 𝜏q ě 𝐶p𝜋𝑡 q ` 𝜆 𝑡 p𝐺p𝜋𝑡 q ´ 𝜏q ´ 𝜖 𝐹𝑄𝐸 p1 ` 𝜆 𝑡 1q p 𝑡 q ` 𝜆J p ě 𝐶p𝜋 𝑡 p𝐺p𝜋𝑡 q ´ 𝜏q ´ 𝜖 𝐹𝑄𝐸 p1 ` 𝐵q p 𝑡 , 𝜆𝑡 q ´ 𝜖 𝐹𝑄𝐸 p1 ` 𝐵q “ 𝐿p𝜋 (A.12) Recall that the execution of mixture policy 𝜋ˆ is done by uniformly sampling one policy 𝜋𝑡 from t𝜋1 , . . . , 𝜋𝑇 u, and rolling-out with 𝜋𝑡 . Thus from equations (A.9) p 𝑡 , 𝜆𝑡 q ď 𝐶p𝜋 ˚ q ` 𝜖 𝐹𝑄𝐼 ` p1 ` 𝐵q𝜖 𝐹𝑄𝐸 w.p. 1 ´ 𝛿. and (A.12), we have E𝑡 „𝑈 r1:𝑇 s 𝐿p𝜋 In other words, with probability 1 ´ 𝛿: 𝑇 1 ÿp 𝐿p𝜋𝑡 , 𝜆𝑡 q ď 𝐶p𝜋 ˚ q ` 𝜖 𝐹𝑄𝐼 ` p1 ` 𝐵q𝜖 𝐹𝑄𝐸 𝑇 𝑡 “1 Due to the no-regret property of our online algorithm (EG in this case): 𝑇 1 ÿp p 𝜋q p 𝜋q p 𝜋, ˆ 𝜆q ´ 𝜔 “ 𝐶p ˆ ` max 𝜆J p𝐺p 𝐿p𝜋𝑡 , 𝜆𝑡 q ě max 𝐿p ˆ ´ 𝜏q ´ 𝜔 𝜆 𝜆 𝑇 𝑡 “1 p 𝜋q´𝜏 If 𝐺p ˆ ď 0 componentwise, choose 𝜆r𝑖s “ 0, 𝑖 “ 1, 2, . . . , 𝑚 and 𝜆r𝑚`1s “ 𝐵. 𝑔𝑖 p𝜋q ˆ ´ 𝜏r𝑖ss and 𝜆r𝑖s “ Otherwise, we can choose 𝜆r 𝑗s “ 𝐵 for 𝑗 “ arg max rp 𝑖 “1:𝑚 `1 0 @𝑖 ‰ 𝑗. We can see that max 𝑚`1 𝜆PR` ,}𝜆}1 “ 𝐵 p 𝜋q 𝜆J p𝐺p ˆ ´ 𝜏q ě 0. Therefore: p 𝜋q 𝐶p ˆ ´ 𝜔 ď 𝐶p𝜋 ˚ q ` 𝜖 𝐹𝑄𝐼 ` p1 ` 𝐵q𝜖 𝐹𝑄𝐸 with probability at least 1 ´ 𝛿 Combined with the first term from equation (A.10): 𝐶p𝜋q ˆ ´ 𝜖 𝐹𝑄𝐸 ´ 𝜔 ď 𝐶p𝜋 ˚ q ` 𝜖 𝐹𝑄𝐼 ` p1 ` 𝐵q𝜖 𝐹𝑄𝐸 or 𝐶p𝜋q ˆ ď 𝐶p𝜋 ˚ q ` 𝜔 ` 𝜖 𝐹𝑄𝐼 ` p2 ` 𝐵q𝜖 𝐹𝑄𝐸 (A.13) We now bring in the generalization error results from our standalone analysis of FQI (appendix A.6) and FQE (appendix A.5) into equation (A.13). Specifically, when ´ ¯ 𝐾 p𝑚 `1q 24¨214¨𝑉s4 320𝑉s2 𝑛 ě log 𝛿 ` dimF log 𝜖 2 ` logp14𝑒pdimF ` 1qq , when FQI 𝜖2 and FQE are run with 𝐾 iterations, we have the guarantee that for any 𝜖 ą 0, 152 with probability at least 1 ´ 𝛿 𝐶p𝜋q ˆ ď 𝐶p𝜋 ˚ q ` 𝜔 ` ¯ 2𝛾 ´a 𝐾 {2 s 𝐶 𝜖 ` 2𝛾 𝑉 ` 𝜇 3 p1 ´ 𝛾q looooooooooooooooomooooooooooooooooon FQI generalization error ˜ ¸ 𝛾 1{2 p2 ` 𝐵q a 𝛾 𝐾 {2 2𝑉s ` 𝐶𝜇 𝜖 ` p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 loooooooooooooooooooooooomoooooooooooooooooooooooon p2` 𝐵qˆ FQE generalization error ¯ p4 ` 𝐵q𝛾 ´a 𝐾 {2 s ď 𝐶p𝜋 q ` 𝜔 ` 𝐶 𝜇 𝜖 ` 2𝛾 𝑉 p1 ´ 𝛾q3 ˚ (A.14) s 𝜔 𝑉s`𝜔 p 𝜋q From lemma A.3.2, 𝐺p ˆ ď 𝜏 ` 2 𝐶` 𝐵 ď 𝜏 ` 2 𝐵 . From equation (A.11), for p 𝑡 q ě 𝐺p𝜋𝑡 q ´ 𝜖 𝐹𝑄𝐸 1 with probability 1 ´ 𝛿. Thus each t=1,2,. . . ,T, we have 𝐺p𝜋 𝑇 ´ ¯ ÿ p p 𝑡q P 𝐺p𝜋q ˆ ě 𝐺p𝜋q ˆ ´ 𝜖 𝐹𝑄𝐸 1 “ Pp𝐺p𝜋 𝑡 “1 ě 𝐺p𝜋𝑡 q ´ 𝜖 𝐹𝑄𝐸 1|𝜋ˆ “ 𝜋𝑡 qPp𝜋ˆ “ 𝜋𝑡 q 1 ě 𝑇p1 ´ 𝛿q “ 1 ´ 𝛿 𝑇 Therefore, we have the following generalization guarantee for the approximate satisfaction of all constraints: ˜ ¸ a 𝑉s ` 𝜔 𝛾 1{2 𝛾 𝐾 {2 𝐺p𝜋q ˆ ď 𝜏`2 ` 𝐶𝜇 𝜖 ` 2𝑉s (A.15) 𝐵 p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 Inequalities (A.14) and (A.15) complete the proof of theorem A.3.1 (and theorem 2.4.4 of the main thesis chapter 2) 153 A.4 Preliminaries to Analysis of Fitted Q Algorithms In this section, we set-up necessary notations and definitions for the theoretical analysis of FQE and FQI. To simplify the presentation, we will focus exclusively on weighted ℓ2 norm for error analysis. With the definitions and assumptions presented in this section, we will present the sample complexity guarantee of Fitted-Q-Evaluation (FQE) in appendix A.5. The proof for FQI will follow similarly in appendix A.6. While it is possible to adapt proofs from related algorithms (Munos and Szepesvári, 2008; Antos et al., 2008b) to analyze FQE and FQI, in the next two sections we show improved convergence rate from 𝑂p𝑛´4 q to 𝑂p𝑛´2 q, where 𝑛 is the number of samples in data set D. To be consistent with the notations in the main thesis chapter 2, we use the convention 𝐶p𝜋q as the value function that denotes long-term accumulated cost, instead of using 𝑉p𝜋q denoting long-term rewards in the traditional RL literature. Our notation for 𝑄 function is similar to the RL literature - the only difference is that the optimal policy minimizes 𝑄p𝑥, 𝑎q instead of maximizing. We denote the bound on the value function as 𝐶s (alternatively if the single timestep cost is bounded by 𝑐s, then 𝐶s “ 1´𝑐s𝛾 ). For simplicity, the standalone analysis of FQE and FQI concerns only with the cost objective 𝑐. Dealing with cost 𝑐 ` 𝜆J 𝑔 offers no extra difficulty - in s that case we simply augment the bound of the value function to 𝑉s “ 𝐶s ` 𝐵𝐺. Bellman operators s ÞÑ BpX ˆ A; 𝐶q s as The Bellman optimality operator T : BpX ˆ A; 𝐶q ż pT𝑄qp𝑥, 𝑎q “ 𝑐p𝑥, 𝑎q ` 𝛾 min 𝑄p𝑥 1 , 𝑎 1 q𝑝p𝑑𝑥 1 |𝑥, 𝑎q 1 X 𝑎 PA (A.16) The optimal value functions are defined as usual by 𝐶 ˚ p𝑥q “ sup 𝐶 𝜋 p𝑥q and 𝜋 𝑄 ˚ p𝑥, 𝑎q “ sup 𝑄 𝜋 p𝑥, 𝑎q @𝑥 P X, 𝑎 P A. 𝜋 s ÞÑ For a given policy 𝜋, the Bellman evaluation operator T𝜋 : BpX ˆ A; 𝐶q s as BpX ˆ A; 𝐶q ż 𝜋 pT 𝑄qp𝑥, 𝑎q “ 𝑐p𝑥, 𝑎q ` 𝛾 𝑄p𝑥 1 , 𝜋p𝑥 1 qq𝑝p𝑑𝑥 1 |𝑥, 𝑎q (A.17) X It is well known that T𝜋 𝑄 𝜋 “ 𝑄 𝜋 , a fixed point of the T𝜋 operator. 154 Data distribution and weighted ℓ2 norm Denote the state-action data generating distribution as 𝜇, induced by some datagenerating (behavior) policy 𝜋D , that is, p𝑥𝑖 , 𝑎𝑖 q „ 𝜇 for p𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐𝑖 q P D. Note that data set D is formed by multiple trajectories generated by 𝜋D . For each p𝑥𝑖 , 𝑎𝑖 q, we have 𝑥𝑖1 „ 𝑝p¨|𝑥𝑖 , 𝑎𝑖 q and 𝑐𝑖 “ 𝑐p𝑥𝑖 , 𝑎𝑖 q. For any (measurable) function 𝑓 : X ˆ A ÞÑ R, define the 𝜇-weighted ℓ2 norm of 𝑓 as } 𝑓 }2𝜇 “ ş ş 2 𝑎q2 𝜇𝑥 p𝑑𝑥q𝜋D p𝑎|𝑑𝑥q. Similarly for any other XˆA 𝑓 p𝑥, 𝑎q 𝜇p𝑑𝑥, 𝑑𝑎q “ XˆA 𝑓 p𝑥, ş state-action distribution 𝜌, } 𝑓 }2𝜌 “ XˆA 𝑓 p𝑥, 𝑎q2 𝜌p𝑑𝑥, 𝑑𝑎q Inherent Bellman error FQE and FQI depend on a chosen function class F to approximate 𝑄p𝑥, 𝑎q. To express how well the Bellman operator T𝑔 can be approximated by a function in the policy class F , when T𝑔 R F , a notion of distance, known as inherent Bellman error was first proposed by (Munos, 2003) and used in the analysis of related ADP algorithms (Munos and Szepesvári, 2008; Munos, 2007; Antos et al., 2008a,b; Lazaric et al., 2010, 2012; Lazaric and Restelli, 2011; Maillard et al., 2010). Definition A.4.1 (Inherent Bellman Error). Given a function class F and a chosen distribution 𝜌, the inherent Bellman error of F is defined as 𝑑 F “ 𝑑pF , TF q “ sup inf } 𝑓 ´ Tℎ} 𝜌 ℎPF 𝑓 PF where }¨} 𝜌 is the 𝜌´weighted ℓ2 norm and T is the Bellman optimality operator defined in (A.16) To analyze FQE, we will form a similar definition for the Bellman evaluation operator Definition A.4.2 (Inherent Bellman Evaluation Error). Given a function class F and a policy 𝜋, the inherent Bellman evaluation error of F is defined as 𝑑 F𝜋 “ 𝑑pF , T𝜋 F q “ sup inf } 𝑓 ´ T𝜋 ℎ} 𝜌 𝜋 ℎPF 𝑓 PF where }¨} 𝜌 𝜋 is the ℓ2 norm weighted by 𝜌 𝜋 . 𝜌 𝜋 is defined as the state-action distribution induced by policy 𝜋, and T𝜋 is the Bellman operator defined in (A.17) Concentrability coefficients Let 𝑃 𝜋 denote the operator acting on 𝑓 : X ˆ A ÞÑ R such that p𝑃 𝜋 𝑓 qp𝑥, 𝑎q “ ş 1 1 1 1 𝜋 X 𝑓 p𝑥 , 𝜋p𝑥 qq𝑝p𝑥 |𝑥, 𝑎q𝑑𝑥 . Acting on 𝑓 (e.g., approximates 𝑄), 𝑃 captures the transition dynamics of taking action 𝑎 and following 𝜋 thereafters. 155 The following definition and assumption are standard in the analysis of related approximate dynamic programming algorithms (Lazaric et al., 2012; Munos and Szepesvári, 2008; Antos et al., 2008a). As approximate value iteration and policy iteration algorithms perform policy update, the new policy at each round will induce a different stationary state-action distribution. One way to quantify the distribution shift is the notion of concentrability coefficient of future state-action distribution, a variant of the notion introduced by (Munos, 2003). Definition A.4.3 (Concentrability coefficient of state-action distribution). Given data generating distribution 𝜇 „ 𝜋D , initial state distribution 𝜒. For 𝑚 ě 0, and an arbitrary sequence of stationary policies t𝜋𝑚 u𝑚 ě1 let › › › 𝑑p𝜒𝑃 𝜋1 𝑃 𝜋2 . . . 𝑃 𝜋𝑚 q › › › 𝛽 𝜇 p𝑚q “ sup › › 𝑑𝜇 𝜋1 ,...,𝜋 𝑚 8 (𝛽 𝜇 p𝑚q “ 8 if the future state distribution 𝜒𝑃 𝜋1 𝑃 𝜋2 . . . 𝑃 𝜋𝑚 is not absolutely continuous w.r.t. 𝜇, i.e, 𝜒𝑃 𝜋1 𝑃 𝜋2 . . . 𝑃 𝜋𝑚 p𝑥, 𝑎q ą 0 for some 𝜇p𝑥, 𝑎q “ 0) Assumption 3. 𝛽 𝜇 “ p1 ´ 𝛾q2 ř 𝑚𝛾 𝑚 ´1 𝛽 𝜇 p𝑚q ă 8 𝑚 ě1 Combination Lock Example. An example of an MDP that violates Assumption 3 is the “combination lock” example proposed by (Koenig and Simmons, 1996). In this finite MDP, we have 𝑁 states X “ t1, 2, . . . , 𝑁u, and 2 actions: going L or R. The initial state is 𝑥0 “ 1. In any state 𝑥, action 𝐿 takes agent back to initial state 𝑥 0 , and action 𝑅 advances the agent to the next state 𝑥 ` 1 in a chain fashion. Suppose that the reward is 0 everywhere except for the very last state 𝑁. One can see that for an MDP such that any behavior policy 𝜋 𝐷 that has a bounded from below probability of taking action 𝐿 from any state 𝑥, i.e., 𝜋 𝐷 p𝐿|𝑥q ě 𝜈 ą 0, then it takes an exponential number of trajectories to learn or evaluate a policy that always takes action 𝑅. In this setting, we can see that the concentration coefficient 𝛽 𝜇 can be designed to be arbitrarily large. Complexity measure of function class F Definition A.4.4 (Random 𝐿 1 Norm Covers). Let 𝜖 ą 0, let F be a set of functions X ÞÑ R, let 𝑥 1𝑛 “ p𝑥1 , . . . , 𝑥 𝑛 q be 𝑛 fixed points in X. Then a collection of functions F𝜖 “ t 𝑓1 , . . . , 𝑓 𝑁 u is an 𝜖-cover of F on 𝑥 1𝑛 if @ 𝑓 P F , D 𝑓 1 P F𝜖 : | 𝑛 𝑛 1ÿ 1 1ÿ 𝑓 p𝑥𝑖 q ´ 𝑓 p𝑥𝑖 q| ď 𝜖 𝑛 𝑖 “1 𝑛 𝑖 “1 156 The empirical covering number, denote by N1 p𝜖, F , 𝑥1𝑛 q, is the size of the smallest 𝜖-cover on 𝑥1𝑛 . Take N1 p𝜖, F , 𝑥1𝑛 q “ 8 if no finite 𝜖-cover exists. Definition A.4.5 (Pseudo-Dimension). A real-valued function class F has pseudodimension dimF defined as the VC dimension of the function class induced by the sub-level set of functions of F . In other words, define function class H “ tp𝑥, 𝑦q ÞÑ signp 𝑓 p𝑥q ´ 𝑦 : 𝑓 P F u, then dimF “ VC-dimensionpHq 157 A.5 Analysis of Fitted Q Evaluation In this section we prove the following statement for Fitted Q Evaluation (FQE). Theorem A.5.1 (Guarantee for FQE - General Case (theorem 2.4.2 in chapter 2)). Under Assumption 3, for 𝜖 ą 0 & 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q s2 s4 Evaluation (Algorithm 3), for 𝑛 “ 𝑂p 𝐶𝜖 2 plog 𝐾𝛿 ` dimF log 𝐶𝜖 2 ` log dimF qq, we have with probability 1 ´ 𝛿: p 𝐶p𝜋q ´ 𝐶p𝜋q ď a ` ˘ 𝛾 1{2 2𝛾 𝐾 {2𝐶s p 𝛽 𝜇 2𝑑 F𝜋 ` 𝜖 ` q. p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 Theorem A.5.2 (Guarantee for FQE - Bellman Realizable Case). Under Assumptions 3-4, for any 𝜖 ą 0, 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q Evaluation s2 ¨𝐶s4 p log 𝐾𝛿 ` dimF log 320𝜖 2𝐶 ` logp14𝑒pdimF ` 1qqq, (Algorithm 3), when 𝑛 ě 24¨214 𝜖2 we have with probability 1 ´ 𝛿: p 𝐶p𝜋q ´ 𝐶p𝜋q ď a 2𝛾 𝐾 {2𝐶s 𝛾 1{2 p 𝛽𝜇 𝜖 ` q p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 We first focus on theorem A.5.2, analyzing FQE assuming a sufficiently rich function class F so that the Bellman evaluation update T𝜋 is closed wrt F (thus inherent Bellman evaluation error is 0). We call this the Bellman evaluation realizability assumption. This assumption simplifies the presentation of our bounds and also simplifies the final error analysis of Algo. 2. After analyzing FQE under this Bellman realizable setting, we will turn to error bound for general, non-realizable setting in theorem A.5.1 (also theorem 2.4.2 in the main body of the thesis chapter 2). The main difference in the non-realizable setting is the appearance of an extra term 𝑑 F𝜋 our final bound. Error bound for single iteration - Bellman realizable case Assumption 4 (Bellman evaluation realizability). We consider function classes F sufficiently rich so that @ 𝑓 , T𝜋 𝑓 P F . We begin with the following result bounding the error for a single iteration of FQE, under “training” distribution 𝜇 „ 𝜋 𝐷 Proposition A.5.3 (Error bound for single iteration). Let the functions in F also be s and let dimF denote the pseudo-dimension of the function class F . bounded by 𝐶, We have with probability at least 1 ´ 𝛿: }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ă 𝜖 158 ´ ¯ ¨𝐶s4 1 320𝐶s2 when 𝑛 ě 24¨214 log ` dim log ` logp14𝑒pdim ` 1qq F F 𝛿 𝜖2 𝜖2 Remark. Note from proposition A.5.3 that the dependence of sample complexity 𝑛 r 12 q, which is better than previously known analysis for Fitted Value here on 𝜖 is 𝑂p 𝜖 Iteration (Munos and Szepesvári, 2008) and FittedPolicyQ (continuous version of r 14 q. The finite sample Fitted Q Iteration (Antos et al., 2008a)) dependence of 𝑂p 𝜖 r 12 q dependence using linear analysis of LSTD (Lazaric et al., 2010) showed an 𝑂p 𝜖 function approximation. Here we prove similar convergence rate for general nonlinear (bounded) function approximators. Proof of Proposition A.5.3. Recall the training target in round 𝑘 is 𝑦𝑖 “ 𝑐𝑖 ` 𝛾𝑄 𝑘 ´1 p𝑥𝑖1 , 𝜋p𝑥𝑖1 qq for 𝑖 “ 1, 2, . . . , 𝑛, and 𝑄 𝑘 P F is the solution to the following regression problem: 𝑛 1ÿ 𝑄 𝑘 “ arg min p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑓 PF 𝑖 “1 Consider random variables p𝑥, 𝑎q „ 𝜇 and 𝑦 “ 𝑐p𝑥, 𝑎q ` 𝛾𝑄 𝑘 ´1 p𝑥 1 , 𝜋p𝑥 1 qq where 𝑥 1 „ 𝑝p¨|𝑥, 𝑎q. By this definition, T𝜋 𝑄 𝑘 ´1 is the regression function that minimizes square loss min E|ℎp𝑥, 𝑎q ´ 𝑦| 2 out of all functions ℎ (not necessarily in F ). ℎ:R𝑋 ˆ 𝐴ÞÑR This is due to pT𝜋 𝑄 𝑘 ´1 qp𝑥, ˜ 𝑎q ˜ “ E r𝑦|𝑥 “ 𝑥, ˜ 𝑎 “ 𝑎s ˜ by definition of the Bellman operator. Consider 𝑄 𝑘 ´1 fixed and we now want to relate the learned function 𝑄 𝑘 over finite set of 𝑛 samples with the regression function over the whole data distribution via uniform deviation bound. We use the following lemma: Lemma A.5.4 ((Györfi et al., 2006), theorem 11.4. Original version (Lee et al., 1996), theorem 3). Consider random vector p𝑋, 𝑌 q and 𝑛 i.i.d samples p𝑋𝑖 , 𝑌𝑖 q. Let 𝑚p𝑥q be the (optimal) regression function under square loss 𝑚p𝑥q “ Er𝑌 |𝑋 “ 𝑥s. Assume |𝑌 | ď 𝐵 a.s. and 𝐵 ď 1. Let F be a set of function 𝑓 : R𝑑 ÞÑ R and let | 𝑓 p𝑥q| ď 𝐵. Then for each 𝑛 ě 1 " P D 𝑓 P F : E| 𝑓 p𝑋q ´ 𝑌 | 2 ´ E|𝑚p𝑋q ´ 𝑌 | 2 𝑛 ˘ 1 ÿ` ´ | 𝑓 p𝑋𝑖 q ´ 𝑌𝑖 | 2 ´ |𝑚p𝑋𝑖 q ´ 𝑌𝑖 | 2 ě 𝑛 𝑖 “1 * ` 2 2 ˘ 𝜖 ¨ 𝛼 ` 𝛽 ` E| 𝑓 p𝑋q ´ 𝑌 | ´ E|𝑚p𝑋q ´ 𝑌 | ˆ ˙ ˆ ˙ 𝜖 2 p1 ´ 𝜖q𝛼𝑛 𝛽𝜖 𝑛 ď 14 sup N1 , F , 𝑥1 exp ´ 20𝐵 214p1 ` 𝜖q𝐵4 𝑥 1𝑛 where 𝛼, 𝛽 ą 0 and 0 ă 𝜖 ă 1{2 159 To apply this lemma, first note that since T𝜋 𝑄 𝑘 ´1 is the optimal regression function1, we have ” ı “ ‰ E 𝜇 p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 “ E 𝜇 p𝑄 𝑘 p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ` T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 ” ı 2 2 𝜋 𝜋 “ E 𝜇 p𝑄 𝑘 p𝑥, 𝑎q ´ T 𝑄 𝑘 ´1 p𝑥, 𝑎qq s ` E 𝜇 rpT 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q thus ‰ “ }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 “ E p𝑄 𝑘 p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎qq2 ‰ ‰ “ “ “ E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 where by definition ż “ 𝜋 2 E p𝑄 𝑘 p𝑥, 𝑎q ´ T 𝑄 𝑘 ´1 p𝑥, 𝑎qq ‰ p𝑄 𝑘 p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎qq2 𝜇p𝑑𝑥, 𝑑𝑎q “ ż “ p𝑄 𝑘 p𝑥, 𝑎q ´ T𝜋 p𝑥, 𝑎qq2 𝜇𝑥 p𝑑𝑥q𝜋D p𝑎|𝑑𝑥q 1 It is easy to see that if 𝑚p𝑥q “ Er𝑦|𝑥s is the regression function then for any function 𝑓 p𝑥q, we have E rp 𝑓 p𝑥q ´ 𝑚p𝑥qqp𝑚p𝑥q ´ 𝑦q “ 0s 160 r𝑘 „ 𝜇 Next, given a fixed data set 𝐷 ( P }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 ą 𝜖 “ * " “ ‰ “ 𝜋 ‰ 2 2 P E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q ´ E pT 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q ą 𝜖 " ‰ ‰ “ “ ď P E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 ˜ ¸ * 𝑛 𝑛 ÿ 1ÿ 1 2 𝜋 2 ´2¨ p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q ´ pT 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q ą 𝜖 𝑛 𝑖 “1 𝑛 𝑖 “1 (A.18) " “ ‰ “ ‰ “ P E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 𝑛 ´ ‰ 1 ÿ“ p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ´ pT𝜋 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑖 “1 * “ ‰ “ 𝜋 ‰ 1 2 2 ą p𝜖 ` E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q ´ E pT 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q q 2 (A.19) " “ ‰ “ ‰ ď P D 𝑓 P F : E p 𝑓 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 𝑛 ´ ‰ 1 ÿ“ p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ´ pT𝜋 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑖 “1 * “ ‰ “ 𝜋 ‰ 1 𝜖 𝜖 2 2 ě p ` ` E p 𝑓 p𝑥, 𝑎q ´ 𝑦q ´ E pT 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q q 2 2 2 ˆ ˙ ˆ ˙ 𝜖 𝑛𝜖 𝑛 ď 14 sup N1 , F , 𝑥1 ¨ exp ´ (A.20) 80𝐶s 24 ¨ 214𝐶s4 𝑥 1𝑛 Equation (A.18) uses the definition of 𝑄 𝑘 “ arg min 𝑛1 𝑓 PF ř𝑛 2 𝑖 “1 p 𝑓 p𝑥𝑖 , 𝑎 𝑖 q ´ 𝑦 𝑖 q and the fact that T𝜋 𝑄 𝑘 ´1 P F , thus making the extra term a positive addition. Equation (A.19) is due to rearranging the terms. Equation (A.20) is an application of lemma A.5.4. We can further bound the empirical covering number by invoking the following lemma due to Haussler (Haussler, 1995): Lemma A.5.5 ((Haussler, 1995), Corollary 3). For any set 𝑋, any points 𝑥 1:𝑛 P s with pseudo-dimension X𝑛 , any class F of functions on 𝑋 taking values in r0, 𝐶s dimF ă 8, and any 𝜖 ą 0 ˆ ˙dim F 2𝑒𝐶s 𝑛 N1 p𝜖, F , 𝑥1 q ď 𝑒pdimF ` 1q 𝜖 161 Applying lemma A.5.5 to equation (A.20), we have the inequality ˆ ˙ ˙dim F ˆ ( 320𝐶s2 𝑛𝜖 2 𝜋 P }𝑄 𝑘 ´ T 𝑄 𝑘 ´1 } 𝜇 ą 𝜖 ď 14¨𝑒¨pdimF `1q ¨exp ´ 𝜖 24 ¨ 214𝐶s4 (A.21) We thus have´that when ¯ ¨𝐶s4 1 320𝐶s2 𝑛 ě 24¨214 log ` logp14𝑒pdim ` 1qq , then: ` dim log F F 𝛿 𝜖2 𝜖2 }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 } 𝜌 ă 𝜖 with probability at least 1 ´ 𝛿. Notice that the dependence r 12 q, which is better than previously known of sample complexity 𝑛 here on 𝜖 is 𝑂p 𝜖 analyses for other approximate dynamic programming algorithms such as Fitted Value Iteration (Munos and Szepesvári, 2008), FittedPolicyQ (Antos et al., 2008b,a) with dependence of 𝑂p 𝜖14 q. Error bound for single iteration - Bellman non-realizable case We now give similar error bound for the general case, where Assumption 4 does not hold. Consider the decomposition “ ‰ “ ‰ }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 “ E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 " “ ‰ “ ‰ “ E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 ˜ ¸* 𝑛 𝑛 ÿ 1ÿ 1 ´2¨ p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ´ pT𝜋 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑖 “1 𝑛 𝑖 “1 ¸* " ˜ ÿ 𝑛 𝑛 ÿ 1 1 p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ´ pT𝜋 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ` 2¨ 𝑛 𝑖 “1 𝑛 𝑖 “1 “ component_1 ` component_2 Splitting the probability of error into two separate bounds. We saw from the previous section (equation (A.21)) that ˆ ˙dim F ˆ ˙ 640𝐶s2 𝑛𝜖 Ppcomponent_1 ą 𝜖{2q ď 14 ¨ 𝑒 ¨ pdimF ` 1q ¨ exp ´ 𝜖 48 ¨ 214𝐶s4 (A.22) 𝜋 We no longer have component_2 ď 0 since T 𝑄 𝑘 ´1 R F . Let 𝑓 ˚ “ arg inf } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 . Since 𝑓 PF ř 𝑄 𝑘 “ arg min 𝑛1 𝑖𝑛“1 p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 , we can upper-bound component_2 by 𝑓 PF ˜ component_2 ď 2 ¨ 𝑛 𝑛 1ÿ ˚ 1ÿ 𝜋 2 p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q ´ pT 𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑖 “1 𝑛 𝑖 “1 ¸ 162 We can treat 𝑓 ˚ as a fixed function, unlike random function 𝑄 𝑘 , and use standard concentration inequalities to bound the empirical average from the expectation. Let random variable 𝑧 “ pp𝑥, 𝑎q, 𝑦q, 𝑧𝑖 “ pp𝑥𝑖 , 𝑎𝑖 q, 𝑦𝑖 q, 𝑖 “ 1, . . . , 𝑛 and let ℎp𝑧q “ p 𝑓 ˚ p𝑥, 𝑎q ´ 𝑦q2 ´ pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 We have |ℎp𝑧q| ď 4𝐶s2 . We will derive a bound for ¸ ˜ 𝑛 1ÿ 𝜖 P ℎp𝑧𝑖 q ´ Eℎp𝑧q ą ` Eℎp𝑧q 𝑛 𝑖 “1 4 using Bernstein inequality(Mohri et al., 2012). First, using the relationship ℎp𝑧q “ p 𝑓 ˚ p𝑥, 𝑎q ` T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 2𝑦qp 𝑓 ˚ p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎qq, the variance of ℎp𝑧q can be bounded by a constant factor of Eℎp𝑧q, since “ ‰ Varpℎp𝑧qq ď Eℎp𝑧q2 ď 16𝐶s2 E p 𝑓 ˚ p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎qq2 ` “ ‰ “ ‰˘ “ 16𝐶s2 E p 𝑓 ˚ p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 “ 16𝐶s2 Eℎp𝑧q (A.23) (A.24) Equation (A.23) stems from T𝜋 𝑄 𝑘 ´1 being the optimal regression function. Now we can apply equation (A.24) and Bernstein inequality to obtain ˜ ¸ 𝑛 1ÿ 𝜖 P ℎp𝑧𝑖 q ´ Eℎp𝑧q ą ` Eℎp𝑧q 𝑛 𝑖 “1 4 ˜ ¸ 𝑛 Varpℎp𝑧qq 1ÿ 𝜖 ďP ℎp𝑧𝑖 q ´ Eℎp𝑧q ą ` 𝑛 𝑖 “1 4 16𝐶s2 ˛ ¨ ´ ¯2 Var 𝑛 4𝜖 ` 16 𝐶s2 ‹ ˚ ´ ¯‚ ď exp ˝´ 2 s Var 2Var ` 2 4𝐶3 4𝜖 ` 16 𝐶s2 ¨ ˛ ´ ¯2 ´ ¯˛ ¨ 𝜖 Var Var 𝑛 4 ` 16𝐶s2 𝑛 4𝜖 ` 16 𝐶s2 ˚ ‹ ‚ ¯´ ¯ ‚ “ exp ˝´ ď exp ˝´ ´ 2 ` 8𝐶s2 8𝐶s2 𝜖 Var s 2 s 32 𝐶 32𝐶 ` 3 3 4 ` 16𝐶s2 ˜ ¸ 1 𝑛𝜖 ¨ 2 ď exp ´ 32 𝐶 s 128 ` 3 Thus ˜ « ff ¸ ˆ ˙ 𝑛 1ÿ 𝜖 3 𝑛𝜖 P 2¨ ℎp𝑧𝑖 q ´ 2Eℎp𝑧q ą ď exp ´ ¨ 𝑛 𝑖 “1 2 416 𝐶s2 (A.25) 163 Now we have « ff 𝑛 𝑛 1ÿ 1ÿ component_2 ď 2 ¨ ℎp𝑧𝑖 q “ 2 ¨ ℎp𝑧𝑖 q ´ 2Eℎp𝑧q ` 4Eℎp𝑧q 𝑛 𝑖 “1 𝑛 𝑖 “1 Using again the fact that T𝜋 𝑄 𝑘 ´1 is the optimal regression function “ ‰ “ ‰ Eℎp𝑧q “ E𝐷 p 𝑓 ˚ p𝑥, 𝑎q ´ 𝑦q2 ´ E𝐷 pT𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 “ ‰ “ E𝐷 p 𝑓 ˚ p𝑥, 𝑎q ´ T𝜋 𝑄 𝑘 ´1 p𝑥, 𝑎qq2 “ inf } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 𝑓 PF (A.26) Combining equations (A.22), (A.25) and (A.26), we can conclude that ( P }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 ´ 4 inf } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 }2𝜇 ą 𝜖 𝑓 PF ˙dim F ˆ ˙ ˆ 𝑛𝜖 640𝐶s2 ¨ exp ´ ď 14 ¨ 𝑒 ¨ pdimF ` 1q 𝜖 48 ¨ 214𝐶s4 ˆ ˙ 3 𝑛𝜖 ¨ ` exp ´ 416 𝐶s2 thus implying ( P }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ´ 2 inf } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ą 𝜖 𝑓 PF ˙ ˙dim F ˆ ˆ 𝑛𝜖 2 640𝐶s2 ¨ exp ´ ď 14 ¨ 𝑒 ¨ pdimF ` 1q 𝜖2 48 ¨ 214𝐶s4 ˆ ˙ 3 𝑛𝜖 2 ` exp ´ ¨ 416 𝐶s2 (A.27) We now can further upper-bound the term 2 inf 𝑓 PF } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ď 2 sup 𝑓 1 PF inf 𝑓 PF } 𝑓 ´ T𝜋 𝑓 1 } 𝜇 “ 2𝑑 F𝜋 (the worst-case inherent Bellman evaluation error), leading to the final bound for the Bellman non-realizable case. One may wish to further remove the inherent Bellman evaluation error from our error bound. However, counter-examples exist where the inherent Bellman error cannot generally be estimated using function approximation (see section 11.6 of (Sutton and Barto, 2018b)). Fortunately, inherent Bellman error can be driven to be small by choosing rich function class F (low bias), at the expense of more samples requirement (higher variance, through higher pseudo-dimension dimF ). While the bound in (A.27) looks more complicated than the Bellman realizable case in equation A.21, note that the convergence rate will still be 𝑂p 𝑛12 q. 164 Bounding the error across iterations Previous sub-sections A.5 and A.5 have analyzed the error of FQE for a single iteration in Bellman realizable and non-realizable case. We now analyze how errors from different iterations flow through the FQE algorithm. The proof borrows the idea from lemma 3 and 4 of (Munos and Szepesvári, 2008) for fitted value iteration (for value function 𝑉 instead of 𝑄), with appropriate modifications for our off-policy evaluation context. Recall that 𝐶 𝜋 , 𝑄 𝜋 denote the true value function and action-value function, respectively, under the evaluation policy 𝜋. And 𝐶𝐾 “ Er𝑄 𝐾 p𝑥, 𝜋p𝑥qqs denote the value function associated with the returned function 𝑄 𝐾 from algorithm 3. Our goal is to bound the difference 𝐶 𝜋 ´ 𝐶𝐾 between the true value function and the estimated value of the returned function 𝑄 𝐾 . Let the unknown state-action distribution induced by the evaluation policy 𝜋 be 𝜌. We first bound the loss }𝑄 𝜋 ´ 𝑄 𝐾 } 𝜌 under the “test-time ”distribution 𝜌 of p𝑥, 𝑎q, which differs from the state-action 𝜇 induced by data-generating policy 𝜋D . We will then lift the loss bound from 𝑄 𝐾 to 𝐶𝐾 . Step 1: Upper-bound the value estimation error s We have for every 𝑘 that Let 𝜖 𝑘 ´1 “ 𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 P X ˆ A, C. 𝑄 𝜋 ´ 𝑄 𝑘 “ T𝜋 𝑄 𝜋 ´ T𝜋 𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 p𝑄 𝜋 is fixed point of 𝑇 𝜋 q “ 𝛾𝑃 𝜋 p𝑄 𝜋 ´ 𝑄 𝑘 ´1 q ` 𝜖 𝑘 ´1 Thus by simple recursion 𝐾ÿ ´1 𝜋 𝑄 ´ 𝑄𝐾 “ 𝛾 𝐾 ´ 𝑘 ´1 p𝑃 𝜋 q𝐾 ´ 𝑘 ´1 𝜖 𝑘 ` 𝛾 𝐾 p𝑃 𝜋 q𝐾 p𝑄 𝜋 ´ 𝑄 0 q 𝑘 “0 𝐾 ´1 1 ´ 𝛾 𝐾 `1 “ ÿ p1 ´ 𝛾q𝛾 𝐾 ´ 𝑘 ´1 𝜋 𝐾 ´ 𝑘 ´1 p𝑃 q 𝜖𝑘 ` “ 𝐾 `1 1´𝛾 1 ´ 𝛾 𝑘 “0 ‰ p1 ´ 𝛾q𝛾 𝐾 𝜋 𝐾 𝜋 p𝑃 q p𝑄 ´ 𝑄 q 0 1 ´ 𝛾 𝐾 `1 « ff 𝐾ÿ ´1 𝐾 ` 1 1´𝛾 “ 𝛼 𝑘 𝐴 𝑘 𝜖 𝑘 ` 𝛼𝐾 𝐴𝐾 p𝑄 𝜋 ´ 𝑄 0 q 1´𝛾 𝑘 “0 where for simplicity of notations, we denote 𝛼𝑘 “ p1 ´ 𝛾q𝛾 𝐾 ´ 𝑘 ´1 p1 ´ 𝛾q𝛾 𝐾 for 𝑘 ă 𝐾, 𝛼 “ 𝐾 1 ´ 𝛾 𝐾 `1 1 ´ 𝛾 𝐾 `1 𝐴 𝑘 “ p𝑃 𝜋 q𝐾 ´ 𝑘 ´1 , 𝐴𝐾 “ p𝑃 𝜋 q𝐾 (A.28) 165 Note that 𝐴 𝑘 ’s are probability kernels and 𝛼 𝑘 ’s are deliberately chosen such that ř 𝑘 𝛼 𝑘 “ 1. We can apply point-wise absolute value on both sides of (A.28) with | 𝑓 | being the short-hand notation for | 𝑓 p𝑥, 𝑎q| and inequality holds point-wise. By triangle inequalities: « ff ´1 𝐾 `1 𝐾ÿ 1 ´ 𝛾 |𝑄 𝜋 ´ 𝑄 𝐾 | ď 𝛼𝑘 𝐴 𝑘 |𝜖 𝑘 | ` 𝛼𝐾 𝐴𝐾 |𝑄 𝜋 ´ 𝑄 0 | (A.29) 1´𝛾 𝑘 “0 Step 2: Bounding }𝑄 𝜋 ´ 𝑄 𝐾 } 𝜌 for any unknown distribution 𝜌. To handle distribution shift from 𝜇 to 𝜌, we decompose the loss as follows: ż 2 𝜋 }𝑄 ´ 𝑄 𝐾 } 𝜌 “ 𝜌p𝑑𝑥, 𝑑𝑎q p𝑄 𝜋 p𝑥, 𝑎q ´ 𝑄 𝐾 p𝑥, 𝑎qq2 «˜ ¸ ff2 2 ż „ 𝐾ÿ ´1 1 ´ 𝛾 𝐾 `1 𝜌p𝑑𝑥, 𝑑𝑎q 𝛼 𝑘 𝐴 𝑘 |𝜖 𝑘 | ` 𝛼𝐾 𝐴𝐾 |𝑄 𝜋 ´ 𝑄 0 | p𝑥, 𝑎q ď 1´𝛾 𝑘 “0 (fromp𝐴.29qq « ff 2 ż „ 𝐾ÿ ´1 1 ´ 𝛾 𝐾 `1 2 ˚ 2 𝜌p𝑑𝑥, 𝑑𝑎q 𝛼 𝑘 p𝐴 𝑘 𝜖 𝑘 q ` 𝛼𝐾 p𝐴𝐾 p𝑄 ´ 𝑄 0 qq p𝑥, 𝑎q ď 1´𝛾 𝑘 “0 (Jensen) « ff „ 2 ż 𝐾ÿ ´1 1 ´ 𝛾 𝐾 `1 ď 𝜌p𝑑𝑥, 𝑑𝑎q 𝛼 𝑘 𝐴 𝑘 𝜖 𝑘2 ` 𝛼𝐾 𝐴𝐾 p𝑄 ˚ ´ 𝑄 0 q2 p𝑥, 𝑎q (Jensen) 1´𝛾 𝑘 “0 Using assumption 3 (assumption 1 of the main thesis chapter 2), we can bound each term 𝜌 𝐴 𝑘 as 𝜌 𝐴 𝑘 “ 𝜌p𝑃 𝜋 q𝐾 ´ 𝑘 ´1 ď 𝜇𝛽 𝜇 p𝐾 ´ 𝑘 ´ 1q (definition A.4.3) Thus }𝑄 𝜋 ´ 𝑄 𝐾 }2𝜌 ď ff „ 2 « 𝐾ÿ ´1 1 ´ 𝛾 𝐾 `1 1 s2 p1 ´ 𝛾q𝛾 𝐾 ´ 𝑘 ´1 𝛽 𝜇 p𝐾 ´ 𝑘 ´ 1q }𝜖 𝑘 }2𝜇 ` 𝛼𝐾 p2𝐶q 1´𝛾 1 ´ 𝛾 𝐾 `1 𝑘 “0 Assumption 3 (stronger than necessary for proof of FQE) can be used to upper-bound the first order concentration coefficient: « ff ÿ ÿ 𝛾 𝛾 p1 ´ 𝛾q 𝛾 𝑚 𝛽 𝜇 p𝑚q ď p1 ´ 𝛾q2 𝑚𝛾 𝑚 ´1 𝛽 𝜇 p𝑚q “ 𝛽𝜇 1´𝛾 1´𝛾 𝑚 ě0 𝑚 ě1 166 This gives the upper-bound for }𝑄 𝜋 ´ 𝑄 𝐾 }2𝜌 as }𝑄 𝜋 ´ 𝑄 𝐾 }2𝜌 ď „ 2 „  p1 ´ 𝛾q𝛾 𝐾 s 2 1 ´ 𝛾 𝐾 `1 𝛾 2 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` p2𝐶q 𝑘 1´𝛾 p1 ´ 𝛾qp1 ´ 𝛾 𝐾 `1 q 1 ´ 𝛾 𝐾 `1  „ 1 ´ 𝛾 𝐾 `1 𝛾 2 𝐾 2 s ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` p1 ´ 𝛾q𝛾 p2𝐶q 𝑘 p1 ´ 𝛾q2 1 ´ 𝛾 𝛾 𝛾𝐾 2 s2 }𝜖 } ď 𝛽 max ` p2𝐶q 𝜇 𝑘 𝜇 3 𝑘 1´𝛾 p1 ´ 𝛾q Using 𝑎 2 ` 𝑏 2 ď p𝑎 ` 𝑏q2 for nonnegative 𝑎, 𝑏, we conclude that ˜ ¸ 𝐾 {2 1{2 a 𝛾 𝛾 }𝑄 𝜋 ´ 𝑄 𝐾 } 𝜌 ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` 2𝐶s 𝑘 p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 (A.30) Step 3: Turning error bound from 𝑄 to |𝐶 𝜋 ´ 𝐶𝐾 | Now we can choose 𝜌 to be the state-action distribution by the evaluation policy 𝜋. The error bound on the value function 𝐶 follows simply by integrating inequality (A.30) over state-action pairs induced by 𝜋. The final error across iterations can be related to individual iteration error by ˜ ¸ 𝐾 { 2 1 { 2 a 𝛾 𝛾 2𝐶s |𝐶 𝜋 ´ 𝐶𝐾 | ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` (A.31) 3 { 2 𝑘 p1 ´ 𝛾q p1 ´ 𝛾q1{2 Finite-sample guarantees for Fitted Q Evaluation Combining results from (A.21), (A.27) and (A.31), we have the final guarantees for FQE under both realizable and general cases. Realizable Case ´ - Proof of theorem 2A.5.2. From (A.21), when ¯ 24¨214¨𝐶s4 𝐾 320𝐶s 𝑛ě log ` dim log ` logp14𝑒pdim ` 1qq , we have }𝜖 𝑘 } 𝜇 ă 𝜖 F F 𝛿 𝜖2 𝜖2 with probability at least 1 ´ 𝛿{𝐾 for any 0 ď 𝑘 ă 𝐾. Thus we conclude that for any 𝜖 ą 0, 0 ă 𝛿 ă 1, after 𝐾 iterations of Fitted Q Evaluation, the value estimate returned by 𝑄 𝐾 satisfies: ˜ ¸ a 𝛾 1{2 𝛾 𝐾 {2 𝜋 |𝐶 ´ 𝐶𝐾 | ď 𝛽𝜇 𝜖 ` 2𝐶s p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 holds with probability 1 ´ 𝛿 when ´ ¯ 24¨214¨𝐶s4 𝐾 320𝐶s2 log ` dim log ` logp14𝑒pdim ` 1qq . This concludes 𝑛 ě F F 𝛿 𝜖2 𝜖2 the proof of theorem A.5.2. 167 Non-realizable Case - Proof of theorem A.5.1 and theorem 2.4.2 of chapter 2. Similarly, from (A.27) we have ( P }𝑄 𝑘 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ´ 2 inf } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ą 𝜖 𝑓 PF ˆ ˙dim F ˆ ˙ 640𝐶s2 𝑛𝜖 2 ď 14 ¨ 𝑒 ¨ pdimF ` 1q ¨ exp ´ 𝜖2 48 ¨ 214𝐶s4 ˙ ˆ 3 𝑛𝜖 2 ` exp ´ ¨ 416 𝐶s2 (A.32) Since inf 𝑓 PF } 𝑓 ´ T𝜋 𝑄 𝑘 ´1 } 𝜇 ď supℎPF inf 𝑓 PF } 𝑓 ´ T𝜋 ℎ} 𝜇 “ 𝑑 F𝜋 (the inherent Bellman evaluation error), similar arguments to the realizable case lead to the conclusion that for any 𝜖 ą 0, 0 ă 𝛿 ă 1, after 𝐾 iterations of FQE: ˜ ¸ 1{2 𝐾 {2 a 𝛾 𝛾 |𝐶 𝜋 ´ 𝐶𝐾 | ď 𝛽 𝜇 p2𝑑 F𝜋 ` 𝜖q ` 2𝐶s p1 ´ 𝛾q3{2 p1 ´ 𝛾q1{2 ˘ ` s4 s2 holds with probability 1 ´ 𝛿 when 𝑛 “ 𝑂 𝐶𝜖 2 plog 𝐾𝛿 ` dimF log 𝐶𝜖 2 ` log dimF q , thus finishes the proof of theorem A.5.1. r 12 q dependency of 𝑛 is significant improvement Note that in both cases, the 𝑂p 𝜖 over previous finite-sample analysis of related approximate dynamic programming algorithms (Munos and Szepesvári, 2008; Antos et al., 2008b,a). This dependency matches that of previous analysis using linear function approximators from (Lazaric et al., 2012, 2010) for LSTD and LSPI algorithms. Here our analysis, using similar assumptions, is applicable for general non-linear, bounded function classes. , which is an improvement over convergence rate of 𝑂p 𝑛14 q in related approximate dynamic programming algorithms (Antos et al., 2008a,b; Munos and Szepesvári, 2008). 168 A.6 Analysis of Fitted Q Iteration Algorithm and Discussion Algorithm 15 Fitted Q Iteration with Function Approximation: FQIp𝑐q (Ernst et al., 2005) Input: Collected data set D “ t𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐𝑖 u𝑖𝑛“1 . Function class F 1: Initialize 𝑄 0 P F randomly 2: for 𝑘 “ 1, 2, . . . , 𝐾 do 3: Compute target 𝑦𝑖 “ 𝑐𝑖 ` 𝛾 min𝑎 𝑄 𝑘 ´1 p𝑥𝑖1 , 𝑎q @𝑖 r 𝑘 “ tp𝑥𝑖 , 𝑎𝑖 q, 𝑦𝑖 u𝑛 4: Build training set D 𝑖 “1 5: Solve a supervisedř learning problem: 𝑄 𝑘 “ arg min 𝑛1 𝑖𝑛“1 p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑓 PF 6: end for Output: 𝜋 𝐾 p¨q “ arg min 𝑄 𝐾 p¨, 𝑎q (greedy policy with respect to the returned 𝑎 function 𝑄 𝐾 ) The analysis of FQI (algorithm 15) follows analogously from the analysis of FQE from the previous section (Appendix A.5). For brevity, we skip certain detailed derivations, especially those that are largely identical to FQE’s analysis. To the best of our knowledge, a finite-sample analysis of FQI with general nonlinear function approximation has not been published (Continuous FQI from (Antos et al., 2008a) is in fact a Fitted Policy Iteration algorithm and is different from algo 15). In principle, one can adapt existing analysis of fitted value iteration (Munos and Szepesvári, 2008) and FittedPolicyQ (Antos et al., 2008b,a) to show that under similar assumptions, among policies greedy w.r.t. functions in F , FQI will find 𝜖´ r 14 q samples. We derive an improved analysis of FQI optimal policy using 𝑛 “ 𝑂p 𝜖 with general non-linear function approximations, with better sample complexity of r 12 q. We note that the appendix of (Lazaric and Restelli, 2011) contains 𝑛 “ 𝑂p 𝜖 an analysis of LinearFQI showing similar rate to ours, albeit with linear function approximators. In this section, we prove the following statement: Theorem A.6.1 (Guarantee for FQI - General Case (theorem 2.4.3 in chapter 2)). Under Assumption 3, for any 𝜖 ą 0, 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q ` s4 ˘ s2 Iteration (algorithm 15), for 𝑛 “ 𝑂 𝐶𝜖 2 plog 𝐾𝛿 ` dimF log 𝐶𝜖 2 ` log dimF q , we have with probability 1 ´ 𝛿: ˘ 2𝛾 `a 𝐶 ˚ ´ 𝐶p𝜋 𝐾 q ď 𝛽 𝜇 p2𝑑 F ` 𝜖q ` 2𝛾 𝐾 {2𝐶s 3 p1 ´ 𝛾q 169 where 𝜋 𝐾 is the policy greedy with respect to the returned function 𝑄 𝐾 , and 𝐶 ˚ is the value of the optimal policy. The key steps to the proof follow similar scheme to the proof of FQE. We first bound the error for each iteration, and then analyze how the errors flow through the algorithm. Single iteration error bound }𝑄 𝑘 ´ T𝑄 𝑘 ´1 } 𝜇 Here 𝜇 is the state-action distribution induced by the data-generating policy 𝜋D . We begin with the decomposition: ‰ ‰ “ “ }𝑄 𝑘 ´ T𝑄 𝑘 ´1 }2𝜇 “ E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 " “ ‰ “ ‰ “ E p𝑄 𝑘 p𝑥, 𝑎q ´ 𝑦q2 ´ E pT𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 ´ ˜ ¸* 𝑛 𝑛 1ÿ 1ÿ 2 2 2¨ p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q ´ pT𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q 𝑛 𝑖 “1 𝑛 𝑖 “1 ¸* " ˜ ÿ 𝑛 𝑛 1 1ÿ 2 2 ` 2¨ p𝑄 𝑘 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q ´ pT𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q 𝑛 𝑖 “1 𝑛 𝑖 “1 “ component_1 ` component_2 For T the Bellman (optimality) operator (equation A.16), T𝑄 𝑘 ´1 is the regression function that minimizes square loss min E|ℎp𝑥, 𝑎q ´ 𝑦| 2 , with the ranℎ:R𝑋 ˆ 𝐴ÞÑR dom variables p𝑥, 𝑎q „ 𝜇 and 𝑦 “ 𝑐p𝑥, 𝑎q ` 𝛾 min𝑎1 𝑄 𝑘 ´1 p𝑥 1 , 𝑎 1 q where 𝑥 1 „ 𝑝p𝑥 1 |𝑥, 𝑎q. Invoking lemma A.5.4 and following the steps similar to equations (A.18),(A.19),(A.20) and (A.21) from appendix A.5, we can bound the first component as ˆ ˙ ˆ ˙dim F 𝑛𝜖 640𝐶s2 ¨ exp ´ Ppcomponent_1 ą 𝜖{2q ď 14 ¨ 𝑒 ¨ pdimF ` 1q 𝜖 48 ¨ 214𝐶s4 (A.33) 2 1 ř𝑛 ˚ Let 𝑓 “ arg inf } 𝑓 ´ T𝑄 𝑘 ´1 } 𝜇 . Since 𝑄 𝑘 “ arg min 𝑛 𝑖“1 p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 , we 𝑓 PF 𝑓 PF can upper-bound component_2 by ˜ ¸ 𝑛 𝑛 ÿ 1ÿ ˚ 1 component_2 ď 2 ¨ p 𝑓 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 ´ pT𝑄 𝑘 ´1 p𝑥𝑖 , 𝑎𝑖 q ´ 𝑦𝑖 q2 𝑛 𝑖 “1 𝑛 𝑖“1 Let random variable 𝑧 “ pp𝑥, 𝑎q, 𝑦q, 𝑧𝑖 “ pp𝑥𝑖 , 𝑎𝑖 q, 𝑦𝑖 q, 𝑖 “ 1, . . . , 𝑛 and let ℎp𝑧q “ p 𝑓 ˚ p𝑥, 𝑎q ´ 𝑦q2 ´ pT𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 170 We have |ℎp𝑧q| ď 4𝐶s2 . We can derive a bound for ` ř ˘ P 𝑛1 𝑖𝑛“1 ℎp𝑧𝑖 q ´ Eℎp𝑧q ą 4𝜖 ` Eℎp𝑧q using Bernstein inequality, similar to equations (A.23) and (A.24) from appendix A.5 to obtain: ˜ « ff ¸ ˙ ˆ 𝑛 ÿ 1 𝜖 3 𝑛𝜖 P 2¨ (A.34) ¨ ℎp𝑧𝑖 q ´ 2Eℎp𝑧q ą ď exp ´ 𝑛 𝑖 “1 2 416 𝐶s2 Now we have « ff 𝑛 𝑛 1ÿ 1ÿ component_2 ď 2 ¨ ℎp𝑧𝑖 q “ 2 ¨ ℎp𝑧𝑖 q ´ 2Eℎp𝑧q ` 4Eℎp𝑧q 𝑛 𝑖 “1 𝑛 𝑖 “1 Since “ ‰ “ ‰ Eℎp𝑧q “ E𝐷r 𝑘 p 𝑓 ˚ p𝑥, 𝑎q ´ 𝑦q2 ´ E𝐷r 𝑘 pT𝑄 𝑘 ´1 p𝑥, 𝑎q ´ 𝑦q2 “ ‰ “ E𝐷r 𝑘 p 𝑓 ˚ p𝑥, 𝑎q ´ T𝑄 𝑘 ´1 p𝑥, 𝑎qq2 “ inf } 𝑓 ´ T𝑄 𝑘 ´1 }2𝜇 𝑓 PF (A.35) Combining equations (A.33), (A.34) and (A.35), we obtain that ( P }𝑄 𝑘 ´ T𝑄 𝑘 ´1 }2𝜇 ´ 4 inf } 𝑓 ´ T𝑄 𝑘 ´1 }2𝜇 ą 𝜖 ď 𝑓 PF ˙dim F ˆ ˙ ˆ 𝑛𝜖 640𝐶s2 ¨ exp ´ 14 ¨ 𝑒 ¨ pdimF ` 1q 𝜖 48 ¨ 214𝐶s4 ˙ ˆ 3 𝑛𝜖 ¨ ` exp ´ 416 𝐶s2 (A.36) Propagation of error bound for }𝑄 ˚ ´ 𝑄 𝜋𝐾 } 𝜌 The analysis of error propagation for FQI is more involved than that of FQE, but the proof largely follows the error propagation analysis in lemma 3 and 4 of (Munos and Szepesvári, 2008) in the fitted value iteration context (for 𝑉 function). We include the 𝑄 function’s (slighly more complicated) derivation here for completeness. Recall that 𝜋 𝐾 is greedy wrt the learned function 𝑄 𝐾 returned by FQI. We aim to bound the difference 𝐶 ˚ ´ 𝐶 𝜋𝐾 between the optimal value function and that 𝜋 𝐾 . For a (to-be-specified) distribution 𝜌 of state-action pairs (different from the data distribution 𝜇), we bound the generalization loss }𝑄 ˚ ´ 𝑄 𝜋𝐾 } 𝜌 Step 1: Upper-bound the propagation error (value). Let 𝜖 𝑘 ´1 “ 𝑄 𝑘 ´ T𝑄 𝑘 ´1 . 171 We have that ˚ ˚ ˚ 𝑄 ˚ ´ 𝑄 𝑘 “ T𝜋 𝑄 ˚ ´ T𝜋 𝑄 𝑘 ´1 ` T𝜋 𝑄 𝑘 ´1 ´ T𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 ˚ ˚ ˚ ď T𝜋 𝑄 ˚ ´ T𝜋 𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 (b/c T𝑄 𝑘 ´1 ě T𝜋 𝑄 𝑘 ´1 ) ˚ “ 𝛾𝑃 𝜋 p𝑄 ˚ ´ 𝑄 𝑘 ´1 q ` 𝜖 𝑘 ´1 Thus by recursion 𝑄 ˚ ´ 𝑄 𝐾 ď ř 𝐾 ´1 𝑘 “0 𝛾 𝐾 ´ 𝑘 ´1 p𝑃 𝜋 ˚ q𝐾 ´ 𝑘 ´1 𝜖 𝑘 `𝛾 𝐾 p𝑃 𝜋 ˚ q𝐾 p𝑄 ˚ ´ 𝑄 q 0 Step 2: Lower-bound the propagation error (value). Similarly 𝑄 ˚ ´ 𝑄 𝑘 “ T𝑄 ˚ ´ T𝜋 𝑘 ´1 𝑄 ˚ ` T𝜋 𝑘 ´1 𝑄 ˚ ´ T𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 ě T𝜋 𝑘 ´1 𝑄 ˚ ´ T𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 (as T𝑄 ˚ ě T𝜋 𝑘 ´1 𝑄 ˚ ) ě T𝜋 𝑘 ´1 𝑄 ˚ ´ T𝜋 𝑘 ´1 𝑄 𝑘 ´1 ` 𝜖 𝑘 ´1 (b/c 𝜋 𝑘 ´1 greedy wrt 𝑄 𝑘 ´1 q “ 𝛾𝑃 𝜋 𝑘 ´1 p𝑄 ˚ ´ 𝑄 𝑘 ´1 q ` 𝜖 𝑘 ´1 And by recursion 𝐾ÿ ´1 ˚ 𝛾 𝐾 ´ 𝑘 ´1 p𝑃 𝜋𝐾 ´1 𝑃 𝜋𝐾 ´2 . . . 𝑃 𝜋 𝑘 `1 q𝜖 𝑘 ` 𝑄 ´ 𝑄𝐾 ě 𝐾 𝑘 “0 𝜋 𝐾 ´1 𝛾 p𝑃 𝑃 𝜋𝐾 ´2 . . . 𝑃 𝜋0 qp𝑄 ˚ ´ 𝑄 0 q Step 3: Upper-bound the propagation error (policy). Beginning with a decomposition of value wrt to policy 𝜋 𝐾 ˚ ˚ ˚ ˚ ˚ 𝑄 ˚ ´ 𝑄 𝜋 𝐾 “ T𝜋 𝑄 ˚ ´ T𝜋 𝑄 𝐾 ` T𝜋 𝑄 𝐾 ´ T𝜋 𝐾 𝑄 𝐾 ` T𝜋 𝐾 𝑄 𝐾 ´ T𝜋 𝐾 𝑄 𝜋 𝐾 ď pT𝜋 𝑄 ˚ ´ T𝜋 𝑄 𝐾 q ` pT𝜋𝐾 𝑄 𝐾 ´ T𝜋𝐾 𝑄 𝜋𝐾 q ˚ p since T𝜋 𝑄 𝐾 ď T𝑄 𝐾 “ T𝜋𝐾 𝑄 𝐾 q ˚ “ 𝛾𝑃 𝜋 p𝑄 ˚ ´ 𝑄 𝐾 q ` 𝛾𝑃 𝜋𝐾 p𝑄 𝐾 ´ 𝑄 𝜋𝐾 q ˚ “ 𝛾𝑃 𝜋 p𝑄 ˚ ´ 𝑄 𝐾 q ` 𝛾𝑃 𝜋𝐾 p𝑄 𝐾 ´ 𝑄 ˚ ` 𝑄 ˚ ´ 𝑄 𝜋𝐾 q ˚ Thus leading to p𝐼 ´ 𝛾𝑃 𝜋𝐾 qp𝑄 ˚ ´ 𝑄 𝜋𝐾 q ď 𝛾p𝑃 𝜋 ´ 𝑃 𝜋𝐾 qp𝑄 ˚ ´ 𝑄 𝐾 q The operator ř p𝐼 ´ 𝛾𝑃 𝜋𝐾 q is invertible and p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 “ 𝑚 ě0 𝛾 𝑚 p𝑃 𝜋𝐾 q𝑚 is monotonic. Thus ˚ 𝑄 ˚ ´ 𝑄 𝜋𝐾 ď 𝛾p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 p𝑃 𝜋 ´ 𝑃 𝜋𝐾 qp𝑄 ˚ ´ 𝑄 𝐾 q ˚ “ 𝛾p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 𝑃 𝜋 p𝑄 ˚ ´ 𝑄 𝐾 q ´ 𝛾p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 𝑃 𝜋𝐾 p𝑄 ˚ ´ 𝑄 𝐾 q (A.37) 172 Applying inequalities from Step 1 and Step 2 to the RHS of (A.37), we have ˚ 𝑄 ´𝑄 𝜋𝐾 ď p𝐼 ´ 𝛾𝑃 𝜋 𝐾 ´1 „ 𝐾ÿ ´1 q 𝛾 𝐾 ´𝑘 ´ ˚ ¯ 𝜋 𝐾 ´𝑘 𝜋 𝐾 𝜋 𝐾 ´1 𝜋 𝑘 `1 p𝑃 q ´𝑃 𝑃 ...𝑃 𝜖𝑘 𝑘 “0 `𝛾 𝐾 `1 ´ 𝜋 ˚ 𝐾 `1 p𝑃 q ´ p𝑃 𝜋𝐾 𝑃 𝜋 𝐾 ´1  ¯ ˚ . . . 𝑃 q p𝑄 ´ 𝑄 0 q 𝜋0 (A.38) Next we apply point-wise absolute value on RHS of (A.38), with |𝜖 𝑘 | being the shorthand notation for |𝜖 𝑘 p𝑥, 𝑎q| point-wise. Using triangle inequalities and rewriting (A.38) in a more compact form ((Munos and Szepesvári, 2008)): « ff ´1 𝐾 `1 q 𝐾ÿ 2𝛾p1 ´ 𝛾 𝛼 𝑘 𝐴 𝑘 |𝜖 𝑘 | ` 𝛼𝐾 𝐴𝐾 |𝑄 ˚ ´ 𝑄 0 | 𝑄 ˚ ´ 𝑄 𝜋𝐾 ď p1 ´ 𝛾q2 𝑘 “0 where 𝛼 𝑘 “ p1´𝛾 q𝛾 𝐾 ´ 𝑘 ´1 p1´𝛾 q𝛾 𝐾 for 𝑘 ă 𝐾, 𝛼 “ and 𝐾 𝐾 ` 1 1´ 𝛾 1´𝛾 𝐾 `1 ” ˚ ı 1´𝛾 𝜋 𝐾 ´1 𝜋 𝐾 ´𝑘 𝜋 𝐾 𝜋 𝐾 ´1 𝜋 𝑘 `1 p𝐼 ´ 𝛾𝑃 q p𝑃 q `𝑃 𝑃 ...𝑃 for 𝑘 ă 𝐾 𝐴𝑘 “ 2 ” ı ˚ 1´𝛾 𝐴𝐾 “ p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 p𝑃 𝜋 q𝐾 `1 ` 𝑃 𝜋𝐾 𝑃 𝜋𝐾 ´1 . . . 𝑃 𝜋0 2 Note that 𝐴 𝑘 ’s are probability kernels that combine the 𝑃 𝜋𝑖 terms and 𝛼 𝑘 ’s are chosen ř such that 𝑘 𝛼 𝑘 “ 1. Step 4: Bounding }𝑄 ˚ ´ 𝑄 𝜋𝐾 }2𝜌 for any test distribution 𝜌. This step handles distribution shift from 𝜇 to 𝜌 (similar to Step 2 from sub-section A.5 of appendix A.5) „ }𝑄 ˚ ´ 𝑄 𝜋𝐾 }2𝜌 ď « ż 𝐾ÿ ´1 𝜌p𝑑𝑥, 𝑑𝑎q 2𝛾p1 ´ 𝛾 𝐾 `1 q p1 ´ 𝛾q2 2 ˆ ff 𝛼 𝑘 𝐴 𝑘 𝜖 𝑘2 ` 𝛼𝐾 𝐴𝐾 p𝑄 ˚ ´ 𝑄 0 q2 p𝑥, 𝑎q (twice Jensen) 𝑘 “0 Using assumption 3 (assumption 1 in chapter 2), each term 𝜌 𝐴 𝑘 is bounded as ” ˚ ı 1´𝛾 𝜌p𝐼 ´ 𝛾𝑃 𝜋𝐾 q´1 p𝑃 𝜋 q𝐾 ´ 𝑘 ` 𝑃 𝜋𝐾 𝑃 𝜋𝐾 ´1 . . . 𝑃 𝜋 𝑘 `1 2 ” ˚ ı 1´𝛾 ÿ 𝑚 “ 𝛾 𝜌p𝑃 𝜋𝐾 q𝑚 p𝑃 𝜋 q𝐾 ´ 𝑘 ` 𝑃 𝜋𝐾 𝑃 𝜋𝐾 ´1 . . . 𝑃 𝜋 𝑘 `1 2 𝑚 ě0 ÿ ď p1 ´ 𝛾q 𝛾 𝑚 𝛽 𝜇 p𝑚 ` 𝐾 ´ 𝑘q𝜇 (def A.4.3) 𝜌 𝐴𝑘 “ 𝑚 ě0 173 Thus 2 2𝛾p1 ´ 𝛾 𝐾 `1 q }𝑄 ˆ (A.39) p1 ´ 𝛾q2 ff « 𝐾ÿ ´1 ÿ 1 s2 p1 ´ 𝛾q2 𝛾 𝑚 `𝐾 ´ 𝑘 ´1 𝛽 𝜇 p𝑚 ` 𝐾 ´ 𝑘q }𝜖 𝑘 }2𝜇 ` 𝛼𝐾 p2𝐶q 𝐾 ` 1 1´𝛾 𝑘 “0 𝑚 ě0 „ 2 „  𝐾 ` 1 2𝛾p1 ´ 𝛾 q p1 ´ 𝛾q𝛾 𝐾 s 2 1 2 ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` p2𝐶q 𝑘 p1 ´ 𝛾q2 1 ´ 𝛾 𝐾 `1 1 ´ 𝛾 𝐾 `1 „ ˚ ´ 𝑄 𝜋𝐾 }2𝜌 ď (assumption 3) 2 „  2𝛾p1 ´ 𝛾 𝐾 `1 q 1 𝛾𝐾 2 2 s ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` p2𝐶q 𝑘 p1 ´ 𝛾q2 1 ´ 𝛾 𝐾 `1 1 ´ 𝛾 𝐾 `1 „ 2 „  2𝛾 2 𝐾 2 s ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` 𝛾 p2𝐶q 𝑘 p1 ´ 𝛾q2 „ Using 𝑎 2 ` 𝑏 2 ď p𝑎 ` 𝑏q2 for nonnegative 𝑎, 𝑏, we thus conclude that ˙ ˆ a 2𝛾 𝐾 {2 s ˚ 𝜋𝐾 }𝑄 ´ 𝑄 } 𝜌 ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` 2𝛾 𝐶 𝑘 p1 ´ 𝛾q2 (A.40) Step 5: Bounding 𝐶 ˚ ´𝐶 𝜋𝐾 Using the performance difference lemma (lemma 6.1 of (Kakade and Langford, 2002), which states that 𝐶 ˚ ´ 𝐶 𝜋𝐾 “ ´ 1´1 𝛾 E𝑥 „𝑑 𝜋𝐾 𝐴˚ r𝑥, 𝑎s. 𝑎 „𝜋 𝐾 We can upper-bound the performance difference of value function as 𝐶 ˚ ´ 𝐶 𝜋𝐾 “ 1 E𝑥 „𝑑 r𝐶 ˚ p𝑥q ´ 𝑄 ˚ p𝑥, 𝑎qs 1 ´ 𝛾 𝑎„𝜋𝜋𝐾𝐾 1 E𝑥 „𝑑 𝜋𝐾 r𝐶 ˚ p𝑥q ´ 𝑄 ˚ p𝑥, 𝜋 𝐾 p𝑥qqs 1´𝛾 1 ď E𝑥 „𝑑 𝜋𝐾 r𝑄 ˚ p𝑥, 𝜋 ˚ p𝑥qq ´ 𝑄 𝐾 p𝑥, 𝜋 ˚ p𝑥qq ` 𝑄 𝐾 p𝑥, 𝜋 𝐾 p𝑥q ´ 𝑄 ˚ p𝑥, 𝜋 𝐾 p𝑥qqs 1´𝛾 “ (greedy) 1 E𝑥 „𝑑 𝜋𝐾 |𝑄 ˚ p𝑥, 𝜋 ˚ p𝑥qq ´ 𝑄 𝐾 p𝑥, 𝜋 ˚ p𝑥qq| ` |𝑄 𝐾 p𝑥, 𝜋 𝐾 p𝑥q ´ 𝑄 ˚ p𝑥, 𝜋 𝐾 p𝑥qq| 1´𝛾 ¯ 1 ´ ˚ 𝜋𝐾 ˚ 𝜋𝐾 }𝑄 ´ 𝑄 }𝑑 𝜋 ˆ𝜋˚ ` }𝑄 ´ 𝑄 }𝑑 𝜋 ˆ𝜋𝐾 ď 𝐾 𝐾 1´𝛾 ď (upper-bound 1-norm by 2-norm) ˆ ˙ a 2𝛾 𝐾 {2 s ď 𝛽 𝜇 max }𝜖 𝑘 } 𝜇 ` 2𝛾 𝐶 𝑘 p1 ´ 𝛾q3 (A.41) ˚ Note that inequality (A.41) follows from (A.40) by specifying 𝜌 “ 𝜒𝑃 𝜋𝐾 𝑃 𝜋 and 𝜌 “ 𝜒𝑃 𝜋𝐾 𝑃 𝜋𝐾 , respectively (𝜒 is the initial state distribution). 174 Finite-sample guarantees for Fitted Q Iteration From (A.36) we have: ( P }𝑄 𝑘 ´ T𝑄 𝑘 ´1 } 𝜇 ´ 2 inf } 𝑓 ´ T𝑄 𝑘 ´1 } 𝜇 ą 𝜖 𝑓 PF ˆ ˙ ˙dim F ˆ 640𝐶s2 𝑛𝜖 2 ď 14 ¨ 𝑒 ¨ pdimF ` 1q ¨ exp ´ 𝜖2 48 ¨ 214𝐶s4 ˆ ˙ 3 𝑛𝜖 2 ` exp ´ ¨ 416 𝐶s2 (A.42) Note that inf 𝑓 PF } 𝑓 ´ T𝑄 𝑘 ´1 } 𝜇 ď supℎPF inf 𝑓 PF } 𝑓 ´ Tℎ} 𝜇 “ 𝑑 F (the inherent Bellman error from equation A.16). Combining with equation (A.41), we have the conclusion that for any 𝜖 ą 0, 0 ă 𝛿 ă 1, after 𝐾 iterations of Fitted Q Iteration, and for 𝜋 𝐾 the greedy policy wrt 𝑄 𝐾 : 𝐶 ˚ ´ 𝐶𝐾𝜋 ď ¯ 2𝛾 ´a 𝐾 {2 s 𝛽 p2𝑑 ` 𝜖q ` 2𝛾 𝐶 𝜇 F p1 ´ 𝛾q3 ` s4 ˘ s2 holds with probability 1 ´ 𝛿 when 𝑛 “ 𝑂 𝐶𝜖 2 plog 𝐾𝛿 ` dimF log 𝐶𝜖 2 ` log dimF q . Note that compared to the Fitted Value Iteration analysis of (Munos and Szepesvári, 2008), our error includes an extra factor 2 for 𝑑 F . Statement for the Bellman-realizable Case To facilitate the end-to-end generalization analysis of theorem 2.4.4 in chapter 2, we include a version of FQI analysis under Bellman-realizable assumption in this section. The theorem is a consequence of previous analysis in this section. Assumption 5 (Bellman evaluation realizability). We consider function classes F sufficiently rich so that @ 𝑓 , T 𝑓 P F . Theorem A.6.2 (Guarantee for FQI - Bellman-realizable Case). Under Assumption 3 and 5, for any 𝜖 ą 0, 𝛿 P p0, 1q, after 𝐾 iterations of Fitted Q Iteration, for 𝑛 ě ` ˘ 320𝐶s2 𝐾 24¨214¨𝐶s4 ` dim log log ` logp14𝑒pdim ` 1qq , we have with probability F F 2 2 𝛿 𝜖 𝜖 1 ´ 𝛿: ˘ 2𝛾 `a 𝐾 {2 s 𝐶 ˚ ´ 𝐶p𝜋 𝐾 q ď 𝛽 𝜖 ` 2𝛾 𝐶 𝜇 p1 ´ 𝛾q3 where 𝜋 𝐾 is the policy greedy with respect to the returned function 𝑄 𝐾 , and 𝐶 ˚ is the value of the optimal policy. 175 A.7 Additional Instantiation of Meta-Algorithm We provide an additional instantiation of the meta-algorithm described in chapter 2, with Online Gradient Descent (OGD) (Zinkevich, 2003) and Least-Squares Policy Iteration (LSPI) (Lagoudakis and Parr, 2003b) as subroutines. Using LSPI requires a feature map 𝜙 such that any state-action pair can be represented by 𝑘 features. The value function is linear in parameters represented by 𝜙. Policy representation is simplified to a weight vector 𝑤 P R 𝑘 . Similar to our main algorithm 2, OGD updates require bounded parameters 𝜆. We thus introduce hyper-parameter 𝐵 as the bound of 𝜆 in ℓ2 norm. The gradient update is projected to the ℓ2 ball when the norm of 𝜆 exceeds 𝐵 (line 15 of algo 16). Algorithm 16 Batch Learning under Constraints using Online Gradient Descent and Least-Squares Policy Iteration Input: Dataset D “ t𝑥𝑖 , 𝑎𝑖 , 𝑥𝑖1 , 𝑐𝑖 , 𝑔𝑖 u𝑖𝑛“1 „ 𝜋D . Online algorithm parameters: ℓ2 norm bound 𝐵, learning rate 𝜂 Input: Number of basis function 𝑘. Basis function 𝜙 (feature map for state-action pairs) 1: Initialize 𝜆 1 “ p0, . . . , 0q P R𝑚 2: for each round 𝑡 do 3: Learn 𝑤 𝑡 Ð LSPIp𝑐 ` 𝜆J // LSPI with cost 𝑐 ` 𝜆J 𝑡 𝑔q 𝑡 𝑔 p 4: Evaluate 𝐶p𝑤 𝑡 q Ð LSTDQp𝑤 𝑡 , 𝑐q // Algo 18 with 𝜋𝑡 , cost 𝑐 p 5: Evaluateř𝐺p𝑤 𝑡 q Ð LSTDQp𝑤 𝑡 , 𝑔q // Algo 18 with 𝜋𝑡 , cost 𝑔 𝑡 1 1 p 𝑡 Ð 𝑡 𝑡 1 “1 𝑤 𝑡 6: 𝑤 ř ř p𝑤 p 𝑡 1 q, 𝐺p p 𝑤 p 𝑡1 q p 𝑡 q Ð 1𝑡 𝑡𝑡 1 “1 𝐶p𝑤 p 𝑡 q Ð 1𝑡 𝑡𝑡 1 “1 𝐺p𝑤 7: 𝐶p ř 𝑡 8: 𝜆p𝑡 Ð 1𝑡 𝑡 1 “1 𝜆𝑡 1 r Ð LSPIp𝑐 ` 𝜆pJ 9: Learn 𝑤 // LSPI with cost 𝑐 ` 𝜆pJ 𝑡 𝑔q 𝑡 𝑔 p p r Ð r 𝑐q, 𝐺p𝑤q r Ð LSTDQp r 𝑔q 10: Evaluate 𝐶p𝑤q 𝑤, ¯ ´ LSTDQp𝑤, J p 𝑤 p𝑤 pmax “ max 𝐶p p 𝑡 q ` 𝜆 p𝐺p p 𝑡 q ´ 𝜏q 11: L 𝜆,}𝜆}2 ď 𝐵 p 𝑤q p r ´ 𝜏q pmin “ 𝐶p r ` 𝜆pJ 12: L 𝑡 p𝐺p𝑤q pmax ´ L pmin ď 𝜔 then 13: if L ` ˘ p 𝑡 i.e., 𝜋p𝑡 p𝑥q “ arg min𝑎PA 𝑤 p 𝑡J 𝜙p𝑥, 𝑎q @𝑥 14: Return 𝜋p𝑡 greedy w.r.t 𝑤 15: end if 𝜆 p 𝑡 q ´ 𝜏qq where projection Pp𝜆q “ 𝐵 16: 𝜆𝑡 `1 “ Pp𝜆𝑡 ´ 𝜂p𝐺p𝜋 maxt 𝐵,}𝜆}2 u 17: end for 176 Algorithm 17 Least-Squares Policy Iteration: LSPIp𝑐q (Lagoudakis and Parr, 2003b) Input: Stopping criterion 𝜖 1: Initialize 𝑤 1 Ð 𝑤 0 2: repeat 3: 𝑤 Ð 𝑤1 4: 𝑤 1 Ð LSTDQp𝑤, 𝑐q 5: until }𝑤 ´ 𝑤 1 } ď 𝜖 ` ˘ Output: Policy weight 𝑤 i.e., 𝜋p𝑥q “ arg min𝑎PA 𝑤 J 𝜙p𝑥, 𝑎q @𝑥 Algorithm 18 LSTDQp𝑤, 𝑐q (Lagoudakis and Parr, 2003b) 1: Initialize à Р0 2: Initialize 𝑏˜ Ð 0 3: for each p𝑥, 𝑎, 𝑥 1 , 𝑐q P D do 4: 𝑎 1 “ arg min𝑎PA 𝑤 J 𝜙p𝑥 1 , 𝑎q ˜ ˜ ` // 𝑘 ˆ 𝑘 matrix // 𝑘 ˆ 1 vector ˘J 5: à Рà ` 𝜙p𝑥, 𝑎q 𝜙p𝑥, 𝑎q ´ 𝛾𝜙p𝑥 1 , 𝑎 1 q 6: 𝑏˜ Ð 𝑏˜ ` 𝜙p𝑥, 𝑎q𝑐 7: end for 8: 𝑤˜ Рô1 𝑏˜ Output: 𝑤˜ 177 A.8 Additional Experimental Details Environment Descriptions and Procedures Figure A.1: Depicting the FrozenLake and CarRacing environments. Frozen Lake. The environment is a 8x8 grid as seen in Figure A.1 (left), based on OpenAi’s FrozenLake-v0. In each episode, the agent starts from 𝑆 and traverse to goal 𝐺. While traversing the grid, the agent must avoid the pre-determined holes denoted by 𝐻. If the agent steps off of the grid, the agent returns to the same grid location. The episode terminates when the agent reaches the goal or falls into a hole. The arrows in Figure A.1 (left) is an example policy returned by our algorithm, showing an optimal route. Denote Xℎ𝑜𝑙𝑒𝑠 as the set of all holes in the grid and X𝑔𝑜𝑎𝑙 “ t𝑥 𝑔𝑜𝑎𝑙 u is a singleton set representing the goal in the grid. The contrained batch policy learning problem is: min 𝜋 PΠ s.t. 𝐶p𝜋q “ ErIp𝑥 1 R X𝑔𝑜𝑎𝑙𝑠 qs “ Pp𝑥 1 R t𝑥 𝑔𝑜𝑎𝑙 uq 1 1 (A.43) 𝐺p𝜋q “ ErIp𝑥 P Xℎ𝑜𝑙𝑒𝑠 qs “ Pp𝑥 P Xℎ𝑜𝑙𝑒𝑠 q ď 𝜏 We collect 5000 trajectories by selecting an action randomly with probability .95 and an action from a DDQN-trained model with probability .05.Furthermore we set 𝐵 “ 30 and 𝜂 “ 50, the hyperparameters of our Exponentiated Gradient subroutine. We set the threshold for the constraint 𝜏 “ .1. Car Racing. The environment is a racetrack as seen in Figure A.1 (right), modified from OpenAi’s CarRacing-v0. In each state, given by the raw pixels, the agent has 12 actions: 𝑎 P 𝐴 “ tp𝑖, 𝑗, 𝑘q|𝑖 P t´1, 0, 1u, 𝑗 P t0, 1u, 𝑘 P t0, .2uu. The action tuple p𝑖, 𝑗, 𝑘q cooresponds to steering angle, amount of gas applied and amount of brake applied, respectively. In each episode, the agent starts at the same point on the track and must traverse over 95% of the track, given by a discretization of 281 tiles. The agent recieves a reward of ` 1000 281 for each unique tile over which the agent drives. The agent receives a penalty of ´.1 per-time step. Our collected 178 dataset takes the form: D “ tp𝑥𝑡 ´6 , 𝑥𝑡 ´3 , 𝑥𝑡 q, 𝑎 𝑡 , p𝑥𝑡 ´3 , 𝑥𝑡 , 𝑥𝑡 `3 q, 𝑐 𝑡 , 𝑔0,𝑡 , 𝑔1,𝑡 u where 𝑥𝑖 denotes the image at timestep 𝑖 and 𝑎 𝑡 is applied 3 times between 𝑥𝑡 and 𝑥𝑡 `3 . This frame-stacking option is common practice in online RL for Atari and video games.In our collected dataset D, the maximum horizon is 469 time steps. The first constraint concerns accumulated number of brakes, a proxy for smooth driving or acceleration. The second constraint concerns how far the agent travels away from the center of the track, given by the Euclidean distance between the agent and the closest point on the center of the track. Let 𝑁𝑡 be the number of tiles that is collected by the agent in time 𝑡. The constrained batch policy learning problem is: 8 ÿ min 𝜋 PΠ Er 𝛾 𝑡 p´ 𝑡 “0 1000 𝑁𝑡 ` .1qs 281 8 ÿ s.t. 𝐺 0 p𝜋q “ Er 𝛾 𝑡 Ip𝑎 𝑡 P A𝑏𝑟𝑎𝑘𝑖𝑛𝑔 qs ď 𝜏0 (A.44) 𝑡 “0 8 ÿ 𝐺 1 p𝜋q “ Er 𝛾 𝑡 𝑑p𝑢 𝑡 , 𝑣 𝑡 qs ď 𝜏1 𝑡 “0 We instatiate our subroutines, FQE and FQI, with multi-layered CNNs. Furthermore we set 𝐵 “ 10 and 𝜂 “ .01, the hyperparameters of our Exponentiated Gradient subroutine. We set the threshold for the constraint to be about 75% of the value exhibited by online RL agent trained by DDQN (Van Hasselt et al., 2016). Figure A.2: (First and Second figures) Result of 2-D grid-search for one-shot, regularized policy learning for LSPI (left) and FQI (right). (Third and Fourth figures) value range of individual policies in our mixtured policy and data generating policy 𝜋 𝐷 for main objective (left) and cost constraint (right) 179 Additional Discussion for the Car Racing Experiment Regularized policy learning and grid-search. We perform grid search over a range of regularization parameters 𝜆 for both Least-Squares Policy Iteration - LSPI ((Lagoudakis and Parr, 2003b)) and Fitted Q Iteration - FQI ((Ernst et al., 2005)). The results, seen from the the first and second plot of Figure A.2, show that one-shot regularized learning has difficulty learning a policy that satisfies both constraints. We augment LSPI with non-linear feature mapping from one of our best performing FQI model (using CNNs representation). While both regularized LSPI and regularized FQI can achieve low main objective cost, the constraint cost values tend to be sensitive with the 𝜆 step. Overall for the whole grid search, about 10% of regularized policies satisfy both constraints, while none of the regularized LSPI policy satisfies both constraints. Mixture policy and de-randomization. As our algorithm returned a mixture policy, it is natural to analyze the performance of individual policies in the mixture. The third and fourth plot from Figure A.2 show the range of performance of individual policy in our mixture (purple band). We compare individual policy return with the stochastic behavior of the data generation policy. Note that our policies satisfy constraints almost always, while the individual policy returned in the mixture also tends to outperform 𝜋 𝐷 with respect to the main objective cost. Off-policy evaluation standalone comparison. Typically, inverse propensity scoring based methods call for stochastic behavior and evaluation policies (Precup et al., 2000; Swaminathan and Joachims, 2015). However in this domain, the evaluation policy and environment are both deterministic, with long horizon (the max horizon is D is 469). Consequently Per-Decision Importance Sampling typically evaluates the policy as 0. In general, off-policy policy evaluation in long-horizon domains is known to be challenging (Liu et al., 2018c; Guo et al., 2017). We augment PDIS by approximating the evaluation policy with a stochastic policy, using a softmin temperature parameter. However, PDIS still largely shows significant errors. For Doubly Robust and Weighted Doubly Robust methods, we train a model of the environment as follows: • a 32-dimensional representation of state input is learned using variational autoencoder. Dimensionality reduction is necessary to aid accuracy, as original state dimension is 96 ˆ 96 ˆ 3 • an LSTM is used to learn the transition dynamics 𝑃p𝑧p𝑥 1 q|𝑧p𝑥q, 𝑎q, where 𝑧p𝑥q 180 is the low-dimensional representation learned from previous step. Technically, using a recurrent neural networks is an augmentation to the dynamical modeling, as true MDPs typically do not require long-term memory • the model is trained separately on a different dataset, collected indendently from the dataset D used for evaluation The architecture of our dynamics model is inspired by recent work in model-based online policy learning (Ha and Schmidhuber, 2018). However, despite our best effort, learning the dynamics model accurately proves highly challenging, as the horizon and dimensionality of this domain are much larger than popular benchmarks in the OPE literature (Jiang and Li, 2016a; Thomas and Brunskill, 2016b; Farajtabar et al., 2018a). The dynamics model has difficulty predicting the future state several time steps away. Thus we find that the long-horizon, model-based estimation component of DR and WDR in this high-dimensional setting is not sufficiently accurate. For future work, a thorough benchmarking of off-policy evaluation methods in high-dimensional domains would be a valuable contribution. 181 Appendix B APPENDIX TO CHAPTER 3 182 B.1 Notations Table B.1: Glossary of terms for Chapter 3 Acronym Term 𝑋, 𝐴, 𝑃, 𝑅 𝛾 𝑑0 𝐷 𝜏 𝑇 𝑁 𝜋𝑏 , 𝜋𝑒 𝑉, 𝑄 𝜌 𝑖𝑗: 𝑗 1 State, Action, Dynamics, Reward Discount Factor Initial State Distribution Dataset Trajectory/Episode Horizon/Episode Length Number of episodes in 𝐷 Behavior Policy, Evaluation Policy Value and Action-Value Function, ex: 𝑉p𝜋𝑒 q, 𝑄p𝑥, 𝜋𝑒 q śminp 𝑗 1 ,𝑇 ´1q 𝜋𝑒 p𝑎𝑡𝑖 | 𝑥𝑡𝑖 q Cumulative Importance Weight, 𝑡 “ 𝑗 𝜋 p 𝑎𝑖 | 𝑥 𝑖 q IPS DM HM IS PDIS WIS PDWIS PDWIS FQE IH Q-Reg MRDR AM 𝑄p𝜆q 𝑅p𝜆q Tree DR WDR MAGIC Graph Graph-MC MC Pix-MC Enduro Graph-POMDP GW Pix-GW Inverse Propensity Scoring Direct Method Hybrid Method Importance Sampling Per-Decision Importance Sampling Weighted Importance Sampling Per-Decision Weighted Importance Sampling Per-Decision Weighted Importance Sampling Fitted Q Evaluation (Le et al., 2019a) Infinite Horizon (Liu et al., 2018b) Q Regression (Farajtabar et al., 2018b) More Robust Doubly Robst (Farajtabar et al., 2018b) Approximate Model (Model Based) 𝑄 𝜋 p𝜆q (Harutyunyan et al., 2016) Retracep𝜆q (Munos et al., 2016) Tree-Backupp𝜆q (Precup et al., 2000) Doubly-Robust (Jiang and Li, 2016b; Dudík et al., 2011a) Weighted Doubly-Robust (Dudík et al., 2011a) (Thomas and Brunskill, 2016a) Graph Environment Graph Mountain Car Environment Mountain Car Environment Pixel-Based Mountain Car Environment Enduro Environment Graph-POMDP Environment Gridworld Environment Pixel-Based Gridworld Environment 𝑏 𝑡 𝑡 183 B.2 Ranking of Methods A method that is within 10% of the method with the lowest Relative MSE is counted as a top method, called Near-top Frequency, and then we aggregate across all experiments. See Table B.2 for a sorted list of how often the methods appear within 10% of the best method. Table B.2: Fraction of time among the top estimators across all experiments Method MAGIC FQE DM FQE IH WDR FQE MAGIC 𝑄 𝜋 p𝜆q WDR 𝑄 𝜋 p𝜆q DM 𝑄 𝜋 p𝜆q DR 𝑄 𝜋 p𝜆q WDR R(𝜆) DR FQE MAGIC R(𝜆) WDR Tree DR R(𝜆) DM R(𝜆) DM Tree MAGIC Tree DR Tree DR MRDR WDR Q-Reg DM AM IS WDR MRDR PDWIS DR Q-Reg MAGIC AM MAGIC MRDR DM MRDR PDIS MAGIC Q-Reg WIS NAIVE DM Q-Reg DR AM WDR AM Near-top Frequency 0.300211 0.236786 0.190275 0.177590 0.173362 0.173362 0.150106 0.135307 0.133192 0.128964 0.107822 0.105708 0.105708 0.097252 0.084567 0.076110 0.073996 0.073996 0.071882 0.065539 0.063425 0.054968 0.046512 0.044397 0.038055 0.033827 0.033827 0.033827 0.027484 0.025370 0.025370 0.019027 0.012685 0.006342 Empirical Support for Decision Tree Guideline Tables B.3-B.10 provide a numerical support for the decision tree in the main paper (Figure 3.2). Each table refers to a child node in the decision tree, ordered from 184 left to right, respectively. For example, Table B.3 refers to the left-most child node (propery specified, short horizon, small policy mismatch) while Table B.10 refers to the right-most child node (misspecified, good representation, long horizon, good 𝜋 𝑏 estimate). Table B.3: Near-top Frequency among the properly specified, short horizon, small policy mismatch experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 4.7% 0.0% 7.8% 40.6% 17.2% 21.9% 15.6% 17.2% 4.7% 4.7% 14.1% 23.4% 20.3% 18.8% 12.5% - 3.1% 6.2% 7.8% 21.9% 20.3% 18.8% 12.5% - 4.7% 4.7% 7.8% 34.4% 14.1% 17.2% 14.1% - IPS IS WIS NAIVE Standard Per-Decision 4.7% 3.1% 1.6% 4.7% 3.1% - Table B.4: Near-top Frequency among the properly specified, short horizon, large policy mismatch experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 20.3% 1.6% 3.1% 35.9% 23.4% 15.6% 21.9% 29.7% 1.6% 1.6% 1.6% 14.1% 14.1% 15.6% 12.5% - 0.0% 3.1% 6.2% 17.2% 20.3% 14.1% 18.8% - 7.8% 1.6% 1.6% 37.5% 23.4% 20.3% 21.9% - IPS IS WIS NAIVE Standard Per-Decision 0.0% 0.0% 3.1% 0.0% 1.6% - 185 Table B.5: Near-top Frequency among the properly specified, long horizon, small policy mismatch experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 6.9% 0.0% 1.4% 50.0% 13.9% 20.8% 2.8% 29.2% 0.0% 1.4% 0.0% 22.2% 12.5% 18.1% 1.4% - 0.0% 1.4% 1.4% 23.6% 11.1% 18.1% 0.0% - 5.6% 1.4% 2.8% 50.0% 9.7% 18.1% 2.8% - IPS IS WIS NAIVE Standard Per-Decision 0.0% 0.0% 5.6% 0.0% 0.0% - Table B.6: Near-top Frequency among the properly specified, long horizon, large policy mismatch, deterministic env/rew experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 3.5% 3.5% 3.5% 15.8% 1.8% 22.8% 3.5% 21.1% 3.5% 1.8% 1.8% 17.5% 3.5% 15.8% 3.5% - 1.8% 0.0% 0.0% 29.8% 0.0% 38.6% 1.8% - 1.8% 0.0% 0.0% 28.1% 0.0% 24.6% 1.8% - IPS IS WIS NAIVE Standard Per-Decision 5.3% 0.0% 0.0% 3.5% 8.8% - 186 Table B.7: Near-top Frequency among the properly specified, long horizon, large policy mismatch, stochastic env/rew experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 14.6% 4.2% 4.2% 31.2% 4.2% 2.1% 4.2% 41.7% 0.0% 2.1% 2.1% 2.1% 6.2% 0.0% 6.2% - 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% - 8.3% 2.1% 0.0% 25.0% 0.0% 2.1% 0.0% - IPS IS WIS NAIVE Standard Per-Decision 25.0% 0.0% 2.1% 4.2% 0.0% - Table B.8: Near-top Frequency among the potentially misspecified, insufficient representation experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 3.9% 0.0% 0.0% 6.9% 13.7% 18.6% 5.9% - 25.5% 15.7% 13.7% - 6.9% 5.9% 24.5% - IPS IS WIS NAIVE Standard Per-Decision 10.8% 9.8% 3.9% 8.8% 13.7% - 187 Table B.9: Near-top Frequency among the potentially misspecified, sufficient representation, poor 𝜋 𝑏 estimate experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 0.0% 0.0% 13.3% 0.0% 16.7% 6.7% 20.0% 0.0% 0.0% 0.0% 6.7% 3.3% 0.0% 0.0% 0.0% - 0.0% 3.3% 0.0% 6.7% 6.7% 0.0% 6.7% - 0.0% 0.0% 0.0% 10.0% 20.0% 3.3% 6.7% - IPS IS WIS NAIVE Standard Per-Decision 3.3% 0.0% 0.0% 0.0% 0.0% - Table B.10: Near-top Frequency among the potentially misspecified, sufficient representation, good 𝜋 𝑏 estimate experiments DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 0.0% 0.0% 0.0% 8.3% 2.8% 5.6% 5.6% 0.0% 0.0% 0.0% 5.6% 8.3% 8.3% 5.6% 8.3% - 0.0% 0.0% 0.0% 25.0% 8.3% 8.3% 16.7% - 2.8% 0.0% 5.6% 11.1% 19.4% 0.0% 5.6% - IPS IS WIS NAIVE Standard Per-Decision 0.0% 0.0% 0.0% 0.0% 0.0% - 188 B.3 Challenging Common Wisdom - Supporting Data The following tables represent the numerical support for how horizon and policy difference affect the performance of the OPE estimators when policy mismatch is held constant. Notice that the policy mismatch for table B.12 and B.13 are identical: ` .124573... ˘100 ` .9 ˘10 “ .1 . What we see here is that despite identical policy mismatch, .1 the longer horizon does not impact the error as much (compared to the baseline, Table B.11) as moving 𝜋 𝑒 to .9, far from .1 and keeping the horizon the same. Table B.11: Graph, relative MSE. 𝑇 “ 10, 𝑁 “ 50, 𝜋 𝑏 p𝑎 “ 0q “ 0.1, 𝜋 𝑒 p𝑎 “ 0q “ 0.1246. Dense rewards. Baseline. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 1.9E-3 2.4E-3 5.8E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 1.6E-3 4.9E-3 4.3E-3 8.9E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - 5.0E-3 4.2E-3 9.4E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - 3.4E-3 4.5E-3 9.2E-3 1.8E-3 1.8E-3 1.8E-3 1.8E-3 - IPS IS WIS NAIVE Standard Per-Decision 5.6E-4 1.4E-3 6.1E-3 8.4E-4 1.4E-3 - Table B.12: Graph, relative MSE. 𝑇 “ 100, 𝑁 “ 50, 𝜋 𝑏 p𝑎 “ 0q “ 0.1, 𝜋 𝑒 p𝑎 “ 0q “ 0.1246. Dense rewards. Increasing horizon compared to baseline, fixed 𝜋 𝑒 . DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 5.6E-2 3.4E-3 1.1E-2 6.0E-2 6.0E-2 6.0E-2 3.4E-1 4.7E-4 5.9E-2 1.1E-1 2.5E-1 6.0E-2 6.0E-2 6.0E-2 7.0E-3 - 5.9E-2 1.2E-1 2.9E-1 6.0E-2 6.0E-2 6.0E-2 1.6E-3 - 5.3E-2 9.2E-2 3.1E-1 6.0E-2 6.0E-2 6.0E-2 2.3E-3 - IPS IS WIS NAIVE Standard Per-Decision 1.7E-2 9.5E-4 5.4E-3 2.5E-3 4.9E-4 - 189 Table B.13: Graph, relative MSE. 𝑇 “ 10, 𝑁 “ 50, 𝜋 𝑏 p𝑎 “ 0q “ 0.1, 𝜋 𝑒 p𝑎 “ 0q “ 0.9. Dense rewards. Increasing 𝜋 𝑒 compared to baseline, fixed horizon. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 6.6E-1 5.4E-1 5.4E-1 6.6E-1 6.7E-1 6.6E-1 6.7E-1 1.4E-2 6.7E-1 6.3E-1 7.3E-1 6.6E-1 6.6E-1 6.6E-1 6.6E-1 - 6.6E-1 1.3E0 2.0E0 6.6E-1 9.3E-1 6.6E-1 9.4E-1 - 6.6E-1 9.3E-1 2.0E0 6.6E-1 1.0E0 6.6E-1 1.0E0 - IPS IS WIS NAIVE Standard Per-Decision 1.0E0 2.0E0 4.0E0 5.4E-1 9.7E-1 - 190 B.4 Methods Below we include a description of each of the methods we tested. Let 𝑇˜ “ 𝑇 ´ 1. Inverse Propensity Scoring (IPS) Methods Table B.14: IPS methods. Dudík et al. (2011b); Jiang and Li (2016a) Standard Per-Decision ř𝑇˜ 𝑡 𝑖“1 𝑁 𝑡“0 𝛾 𝑟 𝑡 𝑖 ř 𝑁 𝜌0:𝑇˜ ř𝑇˜ 𝑡 𝑡“0 𝛾 𝑟 𝑡 𝑖“1 𝑤0:𝑇˜ 𝑖 𝑡 𝜌0:𝑡 𝑡“0 𝛾 𝑁 𝑟 𝑡 𝑖“1 𝑖 ř 𝑁 ř𝑇˜ 𝑡 𝜌0:𝑡 𝑡“0 𝛾 𝑤0:𝑡 𝑟 𝑡 𝑖“1 ř𝑁 IS WIS 𝜌𝑖 ˜ 0:𝑇 ř 𝑁 ř𝑇˜ Table B.14 shows the calculation for the four traditional IPS estimators: 𝑉𝐼𝑆 , 𝑉𝑃𝐷 𝐼𝑆 , 𝑉𝑊 𝐼𝑆 , 𝑉𝑃𝐷𝑊 𝐼𝑆 . In addition, we include the following method as well since it is a Rao-Blackwellization Liu et al. (2018b) of the IPS estimators: Hybrid Methods p an estimate of 𝑄, from Hybrid rely on being supplied an action-value function 𝑄, ř p 𝑎q. Doubly-Robust (DR): p which one can also yield 𝑉p𝑥q “ 𝑎P 𝐴 𝜋p𝑎|𝑥q𝑄p𝑥, Thomas and Brunskill (2016a); Jiang and Li (2016b) 𝑁 1 ÿp 𝑖 𝑉𝐷 𝑅 “ 𝑉p𝑥0 q` 𝑁 𝑖 “1 𝑁 8 1 ÿÿ 𝑡 𝑖 𝑖 p 𝑖 𝑖 p 𝑖 qs 𝛾 𝜌0:𝑡 r𝑟 𝑡 ´ 𝑄p𝑥𝑡 , 𝑎 𝑡 q ` 𝛾𝑉p𝑥 𝑡 `1 𝑁 𝑖 “1 𝑡 “0 Weighted Doubly-Robust (WDR): Thomas and Brunskill (2016a) 𝑁 𝑉𝑊 𝐷 𝑅 “ 1 ÿp 𝑖 𝑉p𝑥 0 q` 𝑁 𝑖 “1 𝑁 ÿ 8 ÿ 𝛾𝑡 𝑖 “1 𝑡 “0 𝑖 𝜌0:𝑡 p 𝑡𝑖 , 𝑎𝑖𝑡 q ` 𝛾𝑉p𝑥 p 𝑖 qs r𝑟 𝑡𝑖 ´ 𝑄p𝑥 𝑡 `1 𝑤 0:𝑡 MAGIC: Thomas and Brunskill (2016a) Given 𝑔𝐽 “ t𝑔𝑖 |𝑖 P 𝐽 Ď N Y t´1uu where 𝑔 𝑗 p𝐷q “ 𝑗 𝑁 ÿ ÿ 𝑖 “1 𝑡 “0 𝛾𝑡 𝑁 𝑖 ÿ 𝜌0:𝑡 𝜌𝑖 p 𝑖 q´ 𝑟 𝑡𝑖 ` 𝛾 𝑗 `1 0:𝑡 𝑉p𝑥 𝑗 `1 𝑤 0:𝑡 𝑤 0:𝑡 𝑖 “1 𝑗 𝑁 ÿ ÿ 𝑡 𝛾p 𝑖 “1 𝑡 “0 𝑖 𝜌0:𝑡 𝑤 0:𝑡 p 𝑡𝑖 , 𝑎𝑖𝑡 q ´ 𝑄p𝑥 𝑖 𝜌0: 𝑇˜ 𝑤 0:𝑇˜ p 𝑡𝑖 qq, 𝑉p𝑥 191 then define 𝑑𝑖𝑠𝑡p𝑦, 𝑍q “ min𝑧P𝑍 |𝑦 ´ 𝑧| and p 𝑏 𝑛 p 𝑗q “ 𝑑𝑖𝑠𝑡p𝑔 𝐽𝑗 p𝐷q, 𝐶 𝐼p𝑔 8 p𝐷q, 0.5qq p 𝑛 p𝑖, 𝑗q “ 𝐶𝑜𝑣p𝑔 𝐽 p𝐷q, 𝑔 𝐽 p𝐷qq Ω 𝑖 𝑗 then, for a |𝐽|´simplex Δ|𝐽 | we can calculate p𝑛 ` p 𝑏p 𝑏𝑇 s𝑥 𝑥p˚ P arg min 𝑥𝑇 rΩ 𝑥 PΔ| 𝐽 | which, finally, yields 𝑉𝑀 𝐴𝐺 𝐼𝐶 “ pp 𝑥 ˚ q𝑇 𝑔𝐽 . MAGIC can be thought of as a weighted average of different blends of the DM and Hybrid. In particular, for some 𝑖 P 𝐽, 𝑔𝑖 represents estimating the first 𝑖 steps of p 𝑉p𝜋 𝑒 q according to DR (or WDR) and then estimating the remaining steps via 𝑄. Hence, 𝑉𝑀 𝐴𝐺 𝐼𝐶 finds the most appropriate set of weights which trades off between using a direct method and a Hybrid. Direct Methods (DM) Model-Based Approximate Model (AM): Jiang and Li (2016b) An approach to model-based value estimation is to directly fit the transition dynamics 𝑃p𝑥𝑡 `1 |𝑥𝑡 , 𝑎 𝑡 q, reward 𝑅p𝑥𝑡 , 𝑎 𝑡 q, and terminal condition 𝑃p𝑥𝑡 `1 P 𝑋𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙 |𝑥𝑡 , 𝑎 𝑡 q of the MDP using some for of maximum likelihood or function approximation. This yields a simulation environment from which one can extract the value of a policy using an average ř over rollouts. Thus, 𝑉p𝜋q “ Er 𝑇𝑡“1 𝛾 𝑡 𝑟p𝑥𝑡 , 𝑎 𝑡 q|𝑥 0 “ 𝑥, 𝑎 0 “ 𝜋p𝑥0 qs where the expectation is over initial conditions 𝑥 „ 𝑑0 and the transition dynamics of the simulator. Model-Free p 𝜃q, parametrized by Every estimator in this section will approximate 𝑄 with 𝑄p¨; p the OPE estimate we seek is some 𝜃. From 𝑄 𝑁 1 ÿÿ p 𝑖 , 𝑎; 𝜃q 𝑉“ 𝜋 𝑒 p𝑎|𝑠q𝑄p𝑠 0 𝑁 𝑖 “1 𝑎 P 𝐴 Note that E𝜋𝑒 𝑄p𝑥𝑡 `1 , ¨q “ ř 𝑎 P 𝐴 𝜋 𝑒 p𝑎|𝑥 𝑡 `1 q𝑄p𝑥 𝑡 `1 , 𝑎q. 192 Direct Model Regression (Q-Reg): Farajtabar et al. (2018b) 𝑁 ÿ 𝑇˜ ´ ¯2 ÿ 1 𝑖 p 𝜃q “ min p 𝑡𝑖 , 𝑎𝑖𝑡 ; 𝜃q 𝑄p¨, 𝛾 𝑡 𝜌0:𝑡 𝑅𝑡:𝑖 𝑇˜ ´ 𝑄p𝑥 𝜃 𝑁 𝑖 “1 𝑡 “0 𝑅𝑡:𝑖 𝑇˜ “ 𝑇˜ ÿ 1 𝛾 𝑡 ´𝑡 𝜌p𝑖 𝑡 `1q:𝑡 1 𝑟 𝑡𝑖1 𝑡 1 “𝑡 p 𝜃q “ lim 𝑘 Ñ8 𝑄 p 𝑘 where Fitted Q Evaluation (FQE): Le et al. (2019a) 𝑄p¨, 𝑁 𝑇˜ 1 ÿÿ p p 𝑄 𝑘 “ min p𝑄 𝑘 ´1 p𝑥𝑡𝑖 , 𝑎𝑖𝑡 ; 𝜃q ´ 𝑦𝑖𝑡 q2 𝜃 𝑁 𝑖 “1 𝑡 “0 p 𝑘 ´1 p𝑥 𝑖 , ¨; 𝜃q 𝑦𝑖𝑡 ” 𝑟 𝑡𝑖 ` 𝛾E𝜋𝑒 𝑄 𝑡 `1 Retrace(𝜆) (R(𝜆)), Tree-Backup (Tree), 𝑄 𝜋 p𝜆q: Munos et al. (2016); Precup et al. p 𝜃q “ lim 𝑘 Ñ8 𝑄 p 𝑘 where (2000); Harutyunyan et al. (2016) 𝑄p¨, p 𝑘 p𝑥, 𝑎; 𝜃q “ 𝑄 p 𝑘 ´1 p𝑥, 𝑎; 𝜃q` 𝑄 ÿ E𝜋 𝑏 r 𝑡 ě0 𝛾𝑡 𝑡 ź 𝑐 𝑠 𝑦 𝑡 |𝑥 0 “ 𝑥, 𝑎 0 “ 𝑎s 𝑠 “1 and p 𝑘 ´1 p𝑥𝑡 `1 , ¨; 𝜃q ´ 𝑄 p 𝑘 ´1 p𝑥𝑡 , 𝑎 𝑡 ; 𝜃q 𝑦 𝑡 “ 𝑟 𝑡 ` 𝛾E𝜋𝑒 𝑄 $ 𝜋 p𝑎 |𝑥 q ’ ’ 𝜆 minp1, 𝜋𝑏𝑒 p𝑎𝑠𝑠 |𝑥𝑠𝑠 q q 𝑅p𝜆q ’ & 𝑐 𝑠 “ 𝜆𝜋 𝑒 p𝑎 𝑠 |𝑥 𝑠 q 𝑇𝑟𝑒𝑒 ’ ’ ’ %𝜆 𝑄 𝜋 p𝜆q More Robust Doubly-Robust (MRDR): Farajtabar et al. (2018b) Given Ω𝜋𝑏 p𝑥q “ 𝑑𝑖𝑎𝑔r1{𝜋 𝑏 p𝑎|𝑥qs𝑎P 𝐴 ´ 𝑒𝑒𝑇 𝑒 “ r1, . . . , 1s𝑇 𝑅𝑡:𝑖 𝑇˜ “ 𝑇˜ ÿ 𝛾 𝑗 ´𝑡 𝜌p𝑖 𝑡 `1q: 𝑗 𝑟p𝑥 𝑖𝑗 , 𝑎𝑖𝑗 q 𝑗 “𝑡 and p 𝑎 1 ; 𝜃qs𝑎1 P 𝐴 𝑞 𝜃 p𝑥, 𝑎, 𝑟q “ 𝑑𝑖𝑎𝑔r𝜋 𝑒 p𝑎 1 |𝑥qs𝑎1 P 𝐴 r𝑄p𝑥, ´ 𝑟r1t𝑎 1 “ 𝑎us𝑎1 P 𝐴 193 where 1 is the indicator function, then 𝑁 𝑇˜ 1 ÿ ÿ 2𝑡 𝑖 2 p 𝑄p¨, 𝜃q “ min 𝛾 p𝜌0:𝑇˜ q ˆ 𝜃 𝑁 𝑖 “1 𝑡 “0 𝜌𝑡𝑖 𝑞 𝜃 p𝑥𝑡𝑖 , 𝑎𝑖𝑡 , 𝑅𝑡:𝑖 𝑇˜ q𝑇 Ω𝜋𝑏 p𝑥𝑡𝑖 q𝑞 𝜃 p𝑥𝑡𝑖 , 𝑎𝑖𝑡 , 𝑅𝑡:𝑖 𝑇˜ q State Density Ratio Estimation (IH): Liu et al. (2018b) 𝑁 ÿ 𝑇˜ ÿ 𝛾 𝑡 𝜔p𝑠𝑖𝑡 q𝜌𝑡:𝑡 𝑟 𝑡𝑖 ř𝑁 ř𝑇˜ 𝑖1 𝑡1 𝑖 “1 𝑡 “0 𝑖 1 “0 𝑡 1 “1 𝛾 𝜔p𝑠𝑡 1 q𝜌𝑡 1 :𝑡 1 ř𝑇 𝛾 𝑡 𝑑 𝜋𝑒 p𝑠𝑖𝑡 q 𝑖 𝜔p𝑠𝑡 q “ lim ř𝑇𝑡 “0 𝑖 𝑡 Ñ8 𝑡 𝑡 “0 𝛾 𝑑 𝜋 𝑏 p𝑠𝑡 q 𝑉𝐼𝐻 “ where 𝜋 𝑏 is assumed to be a fixed data-generating policy, and 𝑑 𝜋 is the distribution of states when executing 𝜋 from 𝑠0 „ 𝑑0 . The details for how to find 𝜔 can be found in Algorithm 1 and 2 of Liu et al. (2018b). B.5 Environments For every environment, we initialize the environment with a fixed horizon length 𝑇. If the agent reaches a goal before 𝑇 or if the episode is not over by step 𝑇, it will transition to an environment-dependent absorbing state where it will stay until time 𝑇. For a high level description of the environment features, see Table 3.1. Environment Descriptions Graph Figure B.1 shows a visualization of the Toy-Graph environment. The graph is initialized with horizon 𝑇 and with absorbing state 𝑥 𝑎𝑏𝑠 “ 2𝑇. In each episode, the agent starts at a single starting state 𝑥 0 “ 0 and has two actions, 𝑎 “ 0 and 𝑎 “ 1. At each time step 𝑡 ă 𝑇, the agent can enter state 𝑥𝑡 `1 “ 2𝑡 ` 1 by taking action 𝑎 “ 0, or 𝑥𝑡 `1 “ 2𝑡 ` 2 by taking action 𝑎 “ 1. If the environment is stochastic, we simulate noisy transitions by allowing the agent to slip into 𝑥𝑡 `1 “ 2𝑡 ` 2 instead of 𝑥𝑡 `1 “ 2𝑡 ` 1 and vice-versa with probability .25. At the final time 𝑡 “ 𝑇, the agent always enters the terminal state 𝑥 𝑎𝑏𝑠 . The reward is `1 if the agent transitions to an odd state, otherwise is ´1. If the environment provides sparse rewards, then 𝑟 “ `1 if 𝑥𝑇 ´1 is odd, 𝑟 “ ´1 if 𝑥𝑇 ´1 is even, otherwise 𝑟 “ 0. Similarly to deterministic rewards, if the environment’s rewards are stochastic, then the reward is 𝑟 „ 𝑁p1, 1q if the agent transitions to an odd state, otherwise 𝑟 „ 𝑁p´1, 1q. If the rewards are 194 R = +1 or R ~ ((+1,1) a=0 a=0 1 a=1 3 a= 2T-3 a=1 -10 0 10 11 0 + R= ABS 1 ABS a=0 a=1 2 a=1 4 2T-2 a= 0 -1 R = −1 or R ~ ((−1,1) Figure B.1: Graph Environment a= 0 a= 1 1 a= 1 0 Figure B.2: Graph-MC Environment Figure B.3: MC Environment, pixel-version. The non-pixel version involves representing the state of the car as the position and velocity. S o=1 o=0 a=0 a=0 1 o=T-1 3 2T-3 0 S ABS a=0 2 a=1 4 2T-2 Figure B.5: GraphPOMDP Environment. Model-Fail Thomas and Brunskill (2016a) is a special case of this environment where T=2. We also extend the environment to arbitrary horizon which makes it a semimdp. S F S S H H H S H S F F S S H H S S Figure B.4: Enduro Environment S S a=1 a=1 S S H F H H H F F H F G Figure B.6: Gridworld environment. Blank spaces indicate areas of a small negative reward, S indicates the starting states, F indicates a field of slightly less negative reward, H indicates a hole of severe penalty, G indicates the goal of positive reward. sparse and stochastic then 𝑟 „ 𝑁p1, 1q if 𝑥𝑇 ´1 is odd, otherwise 𝑟 „ 𝑁p´1, 1q and 𝑟 “ 0 otherwise. Graph-POMDP Figure B.5 shows a visualization of the Graph-POMDP environment. The underlying state structure of Graph-POMDP is exactly the Graph environment. However, the states are grouped together based on a choice of Graph-POMDP horizon length, 𝐻. This parameter groups states into 𝐻 observable states. The agent only is able to observe among these states, and not the underlying MDP structure. ModelFail Thomas and Brunskill (2016a) is a special case of this environment when 𝐻 “ 𝑇 “ 2. 195 Graph Mountain Car (Graph-MC) Figure B.2 shows a visualization of the Toy-MC environment. This environment is a 1-D graph-based simplification of Mountain Car. The agent starts at 𝑥 0 “ 0, the center of the valley and can go left or right. There are 21 total states, 10 to the left of the starting position and 11 to the right of the starting position, and a terminal absorbing state 𝑥 𝑎𝑏𝑠 “ 22. The agent receives a reward of 𝑟 “ ´1 at every timestep. The reward becomes zero if the agent reaches the goal, which is state 𝑥 “ `11. If the agent reaches 𝑥 “ ´10 and continues left then the agent remains in 𝑥 “ ´10. If the agent does not reach state 𝑥 “ `11 by step 𝑇 then the episode terminates and the agent transitions to the absorbing state. Mountain Car (MC) We use the OpenAI version of Mountain Car with a few simplifying modifications Brockman et al. (2016); Sutton and Barto (2018a). The car starts in a valley and has to go back and forth to gain enough momentum to scale the mountain and reach the end goal. The state space is given by the position and velocity of the car. At each time step, the car has the following options: accelerate backwards, forwards or do nothing. The reward is 𝑟 “ ´1 for every time step until the car reaches the goal. While the original trajectory length is capped at 200, we decrease the effective length by applying every action 𝑎 𝑡 five times before observing 𝑥𝑡 `1 . Furthermore, we modify the random initial position from being uniformly between r´.6, ´.4s to being one of t´.6, ´.5, ´.4u, with no velocity. The environment is initialized with a horizon 𝑇 and absorbing state 𝑥 𝑎𝑏𝑠 “ r.5, 0s, position at .5 and no velocity. Pixel-based Mountain Car (Pix-MC) This environment is identical to Mountain Car except the state space has been modified from position and velocity to a pixel based representation of a ball, representing a car, rolling on a hill, see Figure B.3. Each frame 𝑓𝑡 is a 80 ˆ 120 image of the ball on the mountain. One cannot deduce velocity from a single frame, so we represent the state as 𝑥𝑡 “ t 𝑓𝑡 ´1 , 𝑓𝑡 u where 𝑓´1 “ 𝑓0 , the initial state. Everything else is identical between the pixel-based version and the position-velocity version described earlier. 196 Enduro We use OpenAI’s implementation of Enduro-v0, an Atari 2600 racing game. We downsample the image to a grayscale of size (84,84). We apply every action one time and we represent the state as 𝑥𝑡 “ t 𝑓𝑡 ´3 , 𝑓𝑡 ´2 , 𝑓𝑡 ´1 , 𝑓𝑡 u where 𝑓𝑖 “ 𝑓0 , the initial state, for 𝑖 ă 0. See Figure B.4 for a visualization. Gridworld (GW) Figure B.6 shows a visualization of the Gridworld environment. The agent starts at a state in the first row or column (denoted S in the figure), and proceeds through the grid by taking actions, given by the four cardinal directions, for 𝑇 “ 25 timesteps. An agent remains in the same state if it chooses an action which would take it out of the environment. If the agent reaches the goal state 𝐺, in the bottom right corner of the environment, it transitions to a terminal state 𝑥 “ 64 for the remainder of the trajectory and receives a reward of `1. In the grid, there is a field (denoted F) which gives the agent a reward of ´.005 and holes (denoted H) which give ´.5. The remaining states give a reward of ´.01. Pixel-Gridworld (Pixel-GW) This environment is identical to Gridworld except the state space has been modified from position to a pixel based representation of the position: 1 for the agent’s location, 0 otherwise. We use the same policies as in the Gridworld case. B.6 Experimental Setup Description of the policies Graph, Graph-POMDP and Graph-MC use static policies with some probability of going left and another probability of going right, ex: 𝜋p𝑎 “ 0q “ 𝑝, 𝜋p𝑎 “ 1q “ 1 ´ 𝑝, independent of state. We vary 𝑝 in our experiments. GW, Pix-GW, MC, Pixel-MC, and Enduro all use an 𝜖´Greedy policy. In other words, we train a policy 𝑄 ˚ (using value iteration or DDQN) and then vary the deviation away from the policy. Hence 𝜖 ´ 𝐺𝑟𝑒𝑒𝑑𝑦p𝑄 ˚ q implies we follow a mixed policy 𝜋 “ arg max𝑎 𝑄 ˚ p𝑥, 𝑎q with probability 1 ´ 𝜖 and uniform with probability 𝜖. We vary 𝜖 in our experiments. 197 Enumeration of Experiments Graph See Table B.15 for a description of the parameters of the experiment we ran in the Graph Environment. The experiments are the Cartesian product of the table. Table B.15: Graph parameters Parameters 𝛾 N T 𝜋 𝑏 p𝑎 “ 0q 𝜋𝑒 p𝑎 “ 0q Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .98 23:11 t4, 16u t.2, .6u .8 {True, False} {True, False} {True, False} {10 of random(0 : 216 )} Tabular False Graph-POMDP See Table B.16 for a description of the parameters of the experiment we ran in the Graph-POMDP Environment. The experiments are the Cartesian product of the table. Table B.16: Graph-POMDP parameters Parameters 𝛾 N (T,H) 𝜋 𝑏 p𝑎 “ 0q 𝜋𝑒 p𝑎 “ 0q Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .98 28:11 tp2, 2q, p16, 6qu t.2, .6u .8 {True, False} {True, False} {True, False} {10 of random(0 : 216 )} Tabular False 198 Gridworld See Table B.17 for a description of the parameters of the experiment we ran in the Gridworld Environment. The experiments are the Cartesian product of the table. Table B.17: Gridworld parameters Parameters 𝛾 N T 𝜖 ´ Greedy, 𝜋 𝑏 𝜖 ´ Greedy, 𝜋𝑒 Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .98 26:11 25 t.2, .4, .6, .8, 1.u .1 False False False {10 of random(0 : 216 )} Tabular True Pixel-Gridworld (Pix-GW) See Table B.18 for a description of the parameters of the experiment we ran in the Pix-GW Environment. The experiments are the Cartesian product of the table. Table B.18: Pix-GW parameters Parameters 𝛾 N T 𝜖 ´ Greedy, 𝜋 𝑏 𝜖 ´ Greedy, 𝜋𝑒 Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .96 26:9 25 t.2, .4, .6, .8, 1.u .1 {True, False} False False {10 of random(0 : 216 )} NN {True, False} 199 Graph-MC See Table B.19 for a description of the parameters of the experiment we ran in the TMC Environment. The experiments are the Cartesian product of the table. Table B.19: Graph-MC parameters Parameters 𝛾 N T p𝜋 𝑏 p𝑎 “ 0q, 𝜋𝑒 p𝑎 “ 0qq Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .99 27:11 250 tp.45, .45q, p.6, .6q, p.45.6q p.6, .45q, p.8, .2q, p.2, .8qu False False False {10 of random(0 : 216 )} Tabular False Mountain Car (MC) See Table B.20 for a description of the parameters of the experiment we ran in the MC Environment. The experiments are the Cartesian product of the table. Table B.20: MC parameters Parameters 𝛾 N T 𝜖 ´ Greedy, p𝜋 𝑏 , 𝜋𝑒 q Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .99 27:10 250 tp.1, 0q, p1, 0q p1, .1q, p.1, 1qu False False False {10 of random(0 : 216 )} {Tabular, NN} False 200 Pixel-Mountain Car (Pix-MC) See Table B.21 for a description of the parameters of the experiment we ran in the Pix-MC Environment. The experiments are the Cartesian product of the table. Table B.21: Pix-MC parameters Parameters 𝛾 N T 𝜖 ´ Greedy, p𝜋 𝑏 , 𝜋𝑒 q Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .97 512 500 tp.25, 0q, p.1, 0q p.25, .1qu False False False {10 of random(0 : 216 )} {Tabular, NN} False Enduro See Table B.22 for a description of the parameters of the experiment we ran in the Enduro Environment. The experiments are the Cartesian product of the table. Table B.22: Enduro parameters Parameters 𝛾 N T 𝜖 ´ Greedy, p𝜋 𝑏 , 𝜋𝑒 q Stochastic Env Stochastic Rew Sparse Rew Seed ModelType Regress 𝜋 𝑏 .9999 512 500 tp.25, 0q, p.1, 0q p.25, .1qu False False False {10 of random(0 : 216 )} {Tabular, NN} False 201 Representation and Function Class For the simpler environments, we use a tabular representation for all the methods. AM amounts to solving for the transition dynamics, rewards, terminal state, etc. through maximum likelihood. FQE, Retrace(𝜆), 𝑄 𝜋 p𝜆q, and Tree-Backup are all implemented through dynamics programming with Q tables. MRDR and Q-Reg used the Sherman Morrison Sherman and Morrison (1950) method to solve the weighted-least square problem, using a basis which spans a table. In the cases where we needed function approximation, we did not directly fit the dynamics for AM; instead, we fit on the difference in states 𝑇p𝑥 1 ´ 𝑥|𝑥, 𝑎q, which is common practice. For the MC environment, we ran experiments with both a linear and NN function class. In both cases, the representation of the state was not changed and remained [position, velocity]. The NN architecture was dense with [16,8,4,2] as the layers. The layers had relu activations (except the last, with a linear activation) and were all initialized with truncated normal centered at 0 with a standard deviation of 0.1. For the pixel-based environments (MC, Enduro), we use a convolutional NN. The architechure is a layer of size 8 with filter (7,7) and stride 3, followed by maxpooling and a layer of size 16 with filter (3,3) and stride 1, followed by max pooling, flattening and a dense layer of size 256. The final layer is a dense layer with the size of the action space, with a linear activation. The layers had elu activations and were all initialized with truncated normal centered at 0 with a standard deviation of 0.1. The layers also have kernel L2 regularizers with weight 1e-6. When using NNs for the IH method, we used the radial-basis function and a shallow dense network for the kernel and density estimate respectively. Choice of hyperparameters Many methods require selection of convergence criteria, regularization parameters, batch sizes, and a whole host of other hyperparameters. Often there is a tradeoff between computational cost and the accuracy of the method. Hyperparameter search is not feasible in OPE since there is no proper validation (like game score in learning). See Table B.23 for a list of hyperparameters that were chosen for the experiments. 202 Table B.23: Hyperparameters for each model by Environment Method AM Q-Reg FQE IH MRDR 𝑅p𝜆q 𝑄 𝜋 p𝜆q Tree Parameter Graph TMC MC Pix-MC Enduro POMDP GW Pix-GW Max Traj Len NN Fit Epochs NN Batchsize NN Train size NN Val size NN Stop delta Omega regul. NN Fit Epochs NN Batchsize NN Train size NN Val size NN Stop delta Convergence 𝜖 Max Iter NN Batchsize Optimizer Clipnorm Quad. prog. regular. NN Fit Epochs NN Batchsize Omega regul. NN Fit Epochs NN Batchsize NN Train size NN Val size NN Stop delta 𝜆 Convergence 𝜖 Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm 𝜆 Convergence 𝜖 Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm 𝜆 Convergence 𝜖 Max Iter NN Fit Epochs NN Batchsize NN Train Size NN ClipNorm T 1 1e-5 1e-3 1 .9 1e-3 500 .9 1e-3 500 .9 1e-3 500 - T 1 1e-5 1e-3 1 .9 2e-3 500 .9 2e-3 500 .9 2e-3 500 - 50 100 32 .8 .2 1e-4 80 32 .8 .2 1e-4 1e-4 160 32 1. 10001 1024 80 1024 .8 .2 1e-4 .9 1e-3 80 4 .03 1. .9 1e-3 80 4 .03 1. .9 1e-3 80 4 .03 1. 50 100 32 .8 .2 1e-4 80 32 .8 .2 1e-4 1e-4 160 32 1. 10001 128 80 1024 .8 .2 1e-4 - 80 32 .8 .2 1e-4 1e-4 600 32 1. 10001 128 80 1024 .8 .2 1e-4 - T 1 1e-5 1e-3 1 .9 1e-3 500 .9 1e-3 500 .9 1e-3 500 - T 1 4e-4 50 1e-3 1 .9 2e-3 50 .9 2e-3 50 .9 2e-3 50 - T 100 25 .8 .2 1e-4 80 32 .8 .2 1e-4 1e-4 80 32 1. 1001 128 80 32 .8 .2 1e-4 .9 1e-3 80 4 .03 1. .9 1e-3 80 4 .03 1. .9 1e-3 80 4 .03 1. 203 B.7 Additional Supporting Figures for Chapter 3 Figure B.7: Enduro DM vs IPS. 𝜋 𝑏 is a policy that deviates uniformly from a trained policy 25% of the time, 𝜋 𝑒 is a policy trained with DDQN. IH has relatively low error mainly due to tracking the simple average, since the kernel function did not learn useful density ratio. The computational time required to calculate the multi-step rollouts of AM, Retrace(𝜆), 𝑄 𝜋 p𝜆q, Tree-Backup(𝜆) exceeded our compute budget and were thus excluded. Figure B.8: MC comparison. 𝑁 “ 256. 𝜋 𝑏 is a uniform random policy, 𝜋 𝑒 is a policy trained with DDQN 204 Figure B.9: Enduro DM vs HM. 𝜋 𝑏 is a policy that deviates uniformly from a trained policy 25% of the time, 𝜋 𝑒 is a policy trained with DDQN. Figure B.10: Comparison of Direct methods’ performance across horizon and number of trajectories in the Toy-Graph environment. Small policy mismatch under a deterministic environment. Figure B.11: (Graph domain) Comparing DMs across horizon length and number of trajectories. Large policy mismatch and a stochastic environment setting. 205 Figure B.12: Comparing DM to DR in a stochastic environment with large policy mismatch. (Graph) Figure B.13: Comparison between FQE, IH and WIS in a low data regime. For low policy mismatch, IPS is competitive to DM in low data, but as the policy mismatch grows, the top DM outperform. Experiments ran in the Gridworld Environment. Figure B.14: Comparison between IPS methods and IH with dense vs sparse rewards. Per-Decision IPS methods see substantial improvement when the rewards are dense. Experiments ran in the Toy-Graph environment with 𝜋p𝑎 “ 0q “ .6, 𝜋 𝑒 p𝑎 “ 0q “ .8 See Tables B.27, B.28, B.29, B.24, B.25, B.26 206 Figure B.15: Exact vs Estimated 𝜋 𝑏 . Exact 𝜋 𝑏 “ .2´Greedy(optimal), 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld, deterministic) Figure B.16: Exact vs Estimated 𝜋 𝑏 . Exact 𝜋 𝑏 “uniform, 𝜋 𝑒 .1´Greedy(optimal). Min error per class. (Pixel Gridworld, deterministic) “ Figure B.17: Hybrid Method comparison. 𝜋 𝑏 p𝑎 “ 0q “ .2, 𝜋 𝑒 p𝑎 “ 0q “ .8. Min error per class. (Graph-MC) 207 Figure B.18: Hybrid Method comparison. 𝜋 𝑏 p𝑎 “ 0q “ .8, 𝜋 𝑒 p𝑎 “ 0q “ .2. Min error per class. (Graph-MC) Figure B.19: Hybrid Method comparison. 𝜋 𝑏 p𝑎 “ 0q “ .6, 𝜋 𝑒 p𝑎 “ 0q “ .6. Min error per class. (Graph-MC) Figure B.20: Hybrid Method comparison. Exact 𝜋 𝑏 “ .2´Greedy(optimal), 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld) 208 Figure B.21: Hybrid Method comparison. 𝜋 𝑏 “ .8´Greedy(optimal), 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld) Figure B.22: Class comparison with unknown 𝜋 𝑏 . At first, HM underperform DM because 𝜋 𝑏 is more difficult to calculate leading to imprecise importance sampling estimates. Exact 𝜋 𝑏 “ .2´Greedy(optimal), 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) Figure B.23: Class comparison with unknown 𝜋 𝑏 . At first, HM underperform DM because 𝜋 𝑏 is more difficult to calculate leading to imprecise importance sampling estimates. Exact 𝜋 𝑏 “ .6´Greedy(optimal), 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) 209 Figure B.24: Class comparison with unknown 𝜋 𝑏 . At first, HM underperform DM because 𝜋 𝑏 is more difficult to calculate leading to imprecise importance sampling estimates. Exact 𝜋 𝑏 “uniform, 𝜋 𝑒 “ .1´Greedy(optimal). Min error per class. (Pixel Gridworld, stochastic env with .2 slippage) Figure B.25: AM Direct vs Hybrid comparison for AM. (Gridworld) Figure B.26: FQE Direct vs Hybrid comparison. (Gridworld) Figure B.27: MRDR Direct vs Hybrid comparison. (Gridworld) 210 Figure B.28: Q-Reg Direct vs Hybrid comparison. (Gridworld) Figure B.29: 𝑄 𝜋 p𝜆q Direct vs Hybrid comparison. (Gridworld) Figure B.30: Retracep𝜆q Direct vs Hybrid comparison. (Gridworld) Figure B.31: Tree-Backup Direct vs Hybrid comparison. (Gridworld) 211 Figure B.32: DR comparison with 𝜋 𝑏 1.´Greedy(optimal). (Pixel Gridworld) “ .2´Greedy(optimal), 𝜋 𝑒 “ Figure B.33: WDR comparison with 𝜋 𝑏 “ .2´Greedy(optimal), 𝜋 𝑒 “ 1.´Greedy(optimal). (Pixel Gridworld) Figure B.34: MAGIC comparison with 𝜋 𝑏 “ .2´Greedy(optimal), 𝜋 𝑒 “ 1.´Greedy(optimal). (Pixel Gridworld) 212 Figure B.35: DR comparison with 𝜋 𝑏 1.´Greedy(optimal). (Pixel Gridworld) “ .8´Greedy(optimal), 𝜋 𝑒 “ Figure B.36: WDR comparison with 𝜋 𝑏 “ .8´Greedy(optimal), 𝜋 𝑒 “ 1.´Greedy(optimal). (Pixel Gridworld) Figure B.37: MAGIC comparison with 𝜋 𝑏 “ .8´Greedy(optimal), 𝜋 𝑒 “ 1.´Greedy(optimal). (Pixel Gridworld) 213 B.8 Additional Supporting Tables to Chapter 3 Table B.24: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 256, 𝜋 𝑏 p𝑎 “ 0q “ 0.2, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Sparse rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 2.4E-3 9.3E-1 8.6E-1 5.1E-5 1.0E0 5.1E-5 1.0E0 1.5E-1 2.4E1 1.2E1 4.6E0 5.1E-5 9.3E-1 5.1E-5 9.3E-1 - 7.3E0 1.2E1 1.5E2 5.1E-5 2.0E0 5.1E-5 2.0E0 - 3.2E-2 8.4E-1 1.5E2 5.1E-5 2.2E0 5.0E-5 2.2E0 - IPS IS WIS NAIVE Standard Per-Decision 9.3E-1 2.0E0 4.0E0 9.3E-1 2.0E0 - Table B.25: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 512, 𝜋 𝑏 p𝑎 “ 0q “ 0.2, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Sparse rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 1.6E-3 1.7E1 9.5E0 5.0E-6 1.0E0 5.0E-6 1.0E0 8.4E-2 1.8E1 9.2E2 9.6E2 5.0E-6 1.6E1 1.1E-5 1.6E1 - 4.6E0 6.1E2 3.7E2 5.0E-6 1.9E0 5.0E-6 1.9E0 - 2.6E-2 2.4E1 1.3E2 5.0E-6 1.6E0 5.0E-6 1.6E0 - IPS IS WIS NAIVE Standard Per-Decision 1.6E1 1.9E0 3.9E0 1.6E1 1.9E0 - 214 Table B.26: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 1024, 𝜋 𝑏 p𝑎 “ 0q “ 0.2, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Sparse rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 1.8E-3 1.2E3 1.8E4 2.4E-5 1.0E0 2.4E-5 1.0E0 2.6E-2 2.3E3 2.2E3 2.5E4 2.4E-5 1.1E3 2.3E-5 1.1E3 - 2.7E0 2.5E1 1.4E2 2.4E-5 2.5E0 2.4E-5 2.5E0 - 3.2E-3 1.3E3 9.6E2 2.4E-5 1.0E0 2.4E-5 1.0E0 - IPS IS WIS NAIVE Standard Per-Decision 1.1E3 2.5E0 3.9E0 1.1E3 2.5E0 - Table B.27: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 256, 𝜋 𝑏 p𝑎 “ 0q “ 0.6, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Dense rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 5.8E-4 1.8E-2 1.4E-2 1.0E-6 1.0E-6 1.0E-6 3.0E-3 3.4E-4 1.5E-3 2.3E-3 3.7E-2 1.0E-6 1.0E-6 1.0E-6 2.6E-4 - 1.6E-3 9.4E-4 2.5E-2 1.0E-6 1.0E-6 1.0E-6 2.4E-4 - 6.5E-4 9.9E-4 1.7E-2 1.0E-6 1.0E-6 1.0E-6 2.4E-4 - IPS IS WIS NAIVE Standard Per-Decision 2.6E-1 1.6E-2 4.5E-1 2.0E-2 1.8E-3 - 215 Table B.28: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 512, 𝜋 𝑏 p𝑎 “ 0q “ 0.6, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Dense rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 5.0E-5 4.5E-3 5.0E-3 7.7E-7 8.1E-7 7.6E-7 2.2E-3 1.7E-4 3.3E-3 7.7E-5 2.6E-3 7.7E-7 8.0E-7 7.7E-7 1.2E-4 - 3.3E-3 3.3E-5 2.7E-3 7.7E-7 7.9E-7 7.7E-7 1.1E-4 - 1.5E-3 4.5E-5 5.7E-3 7.7E-7 8.1E-7 7.6E-7 1.1E-4 - IPS IS WIS NAIVE Standard Per-Decision 4.0E-2 4.0E-3 4.5E-1 4.7E-3 6.0E-4 - Table B.29: Graph, relative MSE. 𝑇 “ 16, 𝑁 “ 1024, 𝜋 𝑏 p𝑎 “ 0q “ 0.6, 𝜋 𝑒 p𝑎 “ 0q “ 0.8. Dense rewards. DM AM Q-Reg MRDR FQE Rp𝜆q Q 𝜋 p𝜆q Tree IH Hybrid Direct DR WDR MAGIC 4.8E-5 2.1E-3 2.9E-3 2.7E-7 2.9E-7 2.7E-7 1.8E-3 9.6E-5 4.7E-4 1.4E-5 5.0E-4 2.7E-7 2.8E-7 2.7E-7 7.7E-5 - 4.8E-4 1.0E-5 4.6E-4 2.7E-7 2.8E-7 2.7E-7 5.8E-5 - 7.2E-5 2.9E-5 9.6E-4 2.7E-7 2.9E-7 2.7E-7 5.8E-5 - IPS IS WIS NAIVE Standard Per-Decision 1.7E-2 1.8E-3 4.4E-1 2.1E-3 2.9E-4 - 216 Appendix C APPENDIX TO CHAPTER 4 C.1 Detailed Theoretical Analysis and Proofs Proof of lemma 4.5.1 Lemma Statement. (Lemma 4.5.1) For a fixed 𝑥, define 𝜋pr𝑥, 𝑎sq fi 𝜑p𝑎q. If 𝜑 is non-negative and 𝐻-smooth w.r.t. 𝑎., then: @𝑎, 𝑎 1 : ` ›2 ˘2 ` ˘› 𝜑p𝑎q ´ 𝜑p𝑎 1 q ď 6𝐻 𝜑p𝑎q ` 𝜑p𝑎 1 q ›𝑎 ´ 𝑎 1 › . The proof of Lemma 4.5.1 rests on 2 properties of 𝐻-smooth functions (differentiable) in R1 , as stated below Lemma C.1.1 (Self-bounding property of Lipschitz-smooth functions). Let 𝜙 : R Ñ a R be an 𝐻-smooth non-negative function. Then for all 𝑎 P R: |∇𝜙p𝑎q| ď 4𝐻𝜙p𝑎q Proof. By mean value theorem, for any 𝑎, 𝑎 1 we have D 𝜂 P p𝑎, 𝑎 1 q (or p𝑎 1 , 𝑎q) such that 𝜙p𝑎 1 q “ 𝜙p𝑎q ` ∇𝜙p𝜂qp𝑎 1 ´ 𝑎q. Since 𝜙 is non-negative, 0 ď 𝜙p𝑎 1 q “ 𝜙p𝑎q ` ∇𝜙p𝑎qp𝑎 1 ´ 𝑎q ` p∇𝜙p𝜂q ´ ∇𝜙p𝑎qqp𝑎 1 ´ 𝑎q ď 𝜙p𝑎q ` ∇𝜙p𝑎qp𝑎 1 ´ 𝑎q ` 𝐻|𝜂 ´ 𝑎||𝑎 1 ´ 𝑎| ď 𝜙p𝑎q ` ∇𝜙p𝑎qp𝑎 1 ´ 𝑎q ` 𝐻|𝑎 1 ´ 𝑎| 2 Choosing 𝑎 1 “ 𝑎 ´ ∇𝜙p𝑎 q 2𝐻 proves the lemma. Lemma C.1.2 (1-d Case (Srebro et al., 2010)). Let 𝜙 : R Ñ R be an 𝐻-smooth non-negative function. Then for all 𝑎, 𝑎 1 P R: ` 𝜙p𝑎q ´ 𝜙p𝑎 1 q ˘2 ` ˘` ˘2 ď 6𝐻 𝜙p𝑎q ` 𝜙p𝑎 1 q 𝑎 ´ 𝑎 1 Proof. As before, D𝜂 P p𝑎, 𝑎 1 q such that 𝜙p𝑎 1 q ´ 𝜙p𝑎q “ ∇𝜙p𝜂qp𝑎 1 ´ 𝑎q. By assumption of 𝜙, we have |∇𝜙p𝜂q ´ ∇𝜙p𝑎q| ď 𝐻|𝜂 ´ 𝑎| ď 𝐻|𝑎 1 ´ 𝑎|. Thus we have: |∇𝜙p𝜂q| ď |∇𝜙p𝑎q| ` 𝐻|𝑎 ´ 𝑎 1 | Consider two cases: (C.1) 217 Case 1: If |𝑎 ´ 𝑎 1 | ď Thus ` |∇𝜙p𝑎 q | 5𝐻 , then by equation C.1 we have |∇𝜙p𝜂q| ď 6{5|∇𝜙p𝑎q|. ˘2 ˘2 ` 𝜙p𝑎q ´ 𝜙p𝑎 1 q “ p∇𝜙p𝜂qq2 𝑎 ´ 𝑎 1 ˘2 ` 36 p∇𝜙p𝑎qq2 𝑎 ´ 𝑎 1 ď 25 ˘2 ` 144 ď 𝐻𝜙p𝑎q 𝑎 ´ 𝑎 1 25 by lemma C.1.1. Therefore: ` ˘2 ` ˘` ˘2 ˘2 ` 𝜙p𝑎q ´ 𝜙p𝑎 1 q ď 6𝐻𝜙p𝑎q 𝑎 ´ 𝑎 1 ď 6𝐻 𝜙p𝑎q ` 𝜙p𝑎 1 q 𝑎 ´ 𝑎 1 Case 2: If |𝑎 ´ 𝑎 1 | ą again ` |∇𝜙p𝑎 q | 1 5𝐻 , then equation C.1 gives |∇𝜙p𝜂q| ď 6𝐻|𝑎 ´ 𝑎 |. Once 𝜙p𝑎q ´ 𝜙p𝑎 1 q ˘2 ` ˘ ` ˘ “ 𝜙p𝑎q ´ 𝜙p𝑎 1 q ∇𝜙p𝜂q 𝑎 ´ 𝑎 1 ` ˘ ` ˘ ď | 𝜙p𝑎q ´ 𝜙p𝑎 1 q ||∇𝜙p𝜂q|| 𝑎 ´ 𝑎 1 | ` ˘ ´ ` ˘2 ¯ ď | 𝜙p𝑎q ´ 𝜙p𝑎 1 q | 6𝐻 𝑎 ´ 𝑎 1 ` ˘` ˘2 ď 6𝐻 𝜙p𝑎q ` 𝜙p𝑎 1 q 𝑎 ´ 𝑎 1 Proof of Lemma 4.5.1. The extension to the multi-dimensional case is straightforward. For any 𝑎, 𝑎 1 P R 𝑘 , consider the function 𝜙 : R Ñ R such that 𝜙p𝑡q “ 𝜑pp1 ´ 𝑡q𝑎 ` 𝑡𝑎 1 q, then 𝜙 is a differentiable, non-negative function and ∇𝑡 p𝜙p𝑡qq “ x∇𝜑p𝑎 ` 𝑡p𝑎 1 ´ 𝑎qq, 𝑎 1 ´ 𝑎y. Thus: |𝜙1 p𝑡1 q ´ 𝜙1 p𝑡2 q| “ |x∇𝜑p𝑎 ` 𝑡1 p𝑎 1 ´ 𝑎qq´ ∇𝜑p𝑎 ` 𝑡2 p𝑎 1 ´ 𝑎qq, 𝑎 1 ´ 𝑎y| › › › › ď ›∇𝜑p𝑎 ` 𝑡 1 p𝑎 1 ´ 𝑎qq ´ ∇𝜑p𝑎 ` 𝑡 2 p𝑎 1 ´ 𝑎qq›˚ ›𝑎 1 ´ 𝑎 › › ›2 ď 𝐻|𝑡1 ´ 𝑡 2 | ›𝑎 ´ 𝑎 1 › Therefore 𝜙 is an 𝐻 }𝑎 ´ 𝑎 1 }2 -smooth function in R. Apply lemma C.1.2 to 𝜙, we have: › ›2 p𝜙p1q ´ 𝜙p0qq2 ď 6𝐻 ›𝑎 ´ 𝑎 1 › p𝜙p1q ` 𝜙p0qq p1 ´ 0q2 which is the same as p𝜑p𝑎q ´ 𝜑p𝑎 1 qq2 ď 6𝐻p𝜑p𝑎q ` 𝜑p𝑎 1 qq }𝑎 ´ 𝑎 1 }2 218 Proof of lemma 4.5.2 Lemma Statement. (Lemma 4.5.2) Given any starting state 𝑠0 , sequentially execute 𝜋 𝑑𝑒𝑡 and 𝜋 𝑠𝑡𝑜 to obtain two separate trajectories A “ t𝑎 𝑡 u𝑇𝑡“1 and à “ t𝑎˜𝑡 u𝑇𝑡“1 such that 𝑎 𝑡 “ 𝜋 𝑑𝑒𝑡 p𝑠𝑡 q and 𝑎˜ 𝑡 “ 𝜋 𝑠𝑡𝑜 p𝑠˜𝑡 q, where 𝑠𝑡 “ r𝑥𝑡 , 𝑎 𝑡 ´1 s and 𝑠˜𝑡 “ r𝑥𝑡 , 𝑎˜ 𝑡 ´1 s. Assuming the policies are stable as per Condition 1, we have Eà r𝑎˜ 𝑡 s “ 𝑎 𝑡 @𝑡 “ 1, . . . , 𝑇, where the expectation is taken over all random roll-outs of 𝜋 𝑠𝑡𝑜 . Proof. Given a starting state 𝑠0 , we prove by induction that Eà r𝑎˜ 𝑡 s “ 𝑎 𝑡 . It is easily seen that the claim is true for 𝑡 “ 1. Now assuming that Eà r𝑎˜ 𝑡 ´1 s “ 𝑎 𝑡 ´1 . We have Eà r𝑎˜ 𝑡 s “ Eà rEr𝑎˜ 𝑡 |𝑠˜𝑡 ss “ Eà r𝛽 𝜋p ˆ 𝑠˜𝑡 q ` p1 ´ 𝛽q𝜋p𝑠˜𝑡 qs “ 𝛽Eà r𝜋p ˆ 𝑠˜𝑡 qs ` p1 ´ 𝛽qEà r𝜋p𝑠˜𝑡 qs Thus: }Eà r𝑎˜ 𝑡 s ´ 𝑎 𝑡 } “ }Eà r𝑎˜ 𝑡 s ´ 𝛽 𝜋p𝑠 ˆ 𝑡 q ´ p1 ´ 𝛽q𝜋p𝑠𝑡 q} “ }𝛽Eà r𝜋p ˆ 𝑠˜𝑡 qs ` p1 ´ 𝛽qEà r𝜋p𝑠˜𝑡 qs ´ 𝛽 𝜋p𝑠 ˆ 𝑡 q ´ p1 ´ 𝛽q𝜋p𝑠𝑡 q} ď 𝛽 }Eà r𝜋p ˆ 𝑠˜𝑡 qs ´ 𝜋p𝑠 ˆ 𝑡 q} ` p1 ´ 𝛽q }Eà r𝜋p𝑠˜𝑡 qs ´ 𝜋p𝑠𝑡 q} ď 𝛽 }Eà r𝑎˜ 𝑡 ´1 s ´ 𝑎 𝑡 ´1 } ` p1 ´ 𝛽q }Eà r𝑎˜ 𝑡 ´1 s ´ 𝑎 𝑡 ´1 } “0 per inductive hypothesis. Therefore we conclude that Eà r𝑎˜ 𝑡 s “ 𝑎 𝑡 @𝑡 “ 1, . . . , 𝑇 Proof of theorem 4.5.6 and corollary 4.5.7 - Main policy improvement results In this section, we provide the proof to theorem 4.5.6 and corollary 4.5.7. Theorem Statement. (theorem 4.5.6) Assume ℓ is convex and 𝐿-Lipschitz-continuous, and Condition 2 holds. Let 𝜖 “ max }𝜋p𝑠q ˆ ´ 𝜋p𝑠q}. Then for 𝛽 P p0, 1q: 𝑠„𝑑 𝜋 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝛾𝜖 𝐿 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq. p1 ´ 𝛽qp1 ´ 𝛾q 219 Proof. First let’s review the notations: let 𝑇 be the trajectory horizon. For a policy 𝜋 in the deterministic policy class Π, given a starting state 𝑠0 , we roll out the full 𝜋 𝜋 𝜋 trajectory 𝑠0 Ý Ñ 𝑠1 Ý Ñ ... Ý Ñ 𝑠𝑇 , where 𝑠𝑡 “ r𝑥𝑡 , 𝜋p𝑠𝑡 ´1 qs, with 𝑥𝑡 encodes the featurized input at current time 𝑡, and 𝜋p𝑠𝑡 ´1 q encodes the dependency on previous predictions. Let ℓp𝜋p𝑠qq be the loss of taking action 𝜋p𝑠q at state 𝑠, we can define the trajectory loss of policy 𝜋 from starting state 𝑠0 as 𝑇 ℓp𝜋|𝑠0 q “ 1ÿ ℓp𝜋p𝑠𝑡 qq 𝑇 𝑡 “1 For a starting state distribution 𝜇, we define policy loss of 𝜋 as the expected loss along trajectories induced by 𝜋: ℓ𝜋 p𝜋q “ E𝑠0 „ 𝜇 rℓp𝜋|𝑠0 qs. Policy loss ℓ𝜋 p𝜋q can be understood as ff « ż 𝑇 1 ÿ ℓp𝜋p𝑠𝑡 qq 𝑑𝜇p𝑠0 q ℓ𝜋 p𝜋q “ E 𝑥 𝑡 „X 𝑇 𝑡 “1 𝑠0 „ 𝜇 To prove policy improvement, we skip the subscript of algorithm 4 to consider general policy update rule within each iteration: 𝜋 1 “ 𝜋𝑛𝑒𝑤 “ 𝛽 𝜋ˆ ` p1 ´ 𝛽q𝜋 (C.2) where 𝜋 “ 𝜋 𝑜𝑙𝑑 is the current policy (combined up until the previous iteration), 𝜋ˆ is p the trained model from calling the base regression routine TrainpS, A|ℎq. Learning rate (step-size) 𝛽 may be adaptively chosen in each iteration. Recall that this update rule reflects deterministic interpolation of two policies. We are interested in quantifying the policy improvement when updating 𝜋 to 𝜋 1 . Specifically, we want to bound Γ “ ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q where ℓ𝜋 p𝜋q (respectively ℓ𝜋1 p𝜋 1 q) denotes the trajectory loss of 𝜋 (respectively 𝜋 1 ) on the state distribution induced by 𝜋 (resp. 𝜋 1 ) We will bound the loss difference of old and new policies conditioned on a common starting state 𝑠0 . Based on update rule (C.2), consider rolling out 𝜋 1 and 𝜋 from the same starting state 𝑠0 to obtain two separate sequences 𝜋 1 ÞÝÑ t𝑠0 Ñ 𝑠11 . . . Ñ 𝑠𝑇1 u and 𝜋 ÞÝÑ t𝑠0 Ñ 𝑠1 . . . Ñ 𝑠𝑇 u corresponding to the same stream of inputs 220 𝑥 1 , . . . , 𝑥𝑇 . 𝑇 1ÿ Γp𝑠0 q “ ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋p𝑠𝑡 qq 𝑇 𝑡 “1 𝑇 1ÿ “ ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋 1 p𝑠𝑡 qq ` ℓp𝜋 1 p𝑠𝑡 qq ´ ℓp𝜋p𝑠𝑡 qq 𝑇 𝑡 “1 (C.3) Assume convexity of ℓ (e.g. sum of square losses): ℓp𝜋 1 p𝑠𝑡 qq “ ℓp𝛽 𝜋p𝑠 ˆ 𝑡 q ` p1 ´ 𝛽q𝜋p𝑠𝑡 qq ď 𝛽ℓp𝜋p𝑠 ˆ 𝑡 qq ` p1 ´ 𝛽qℓp𝜋p𝑠𝑡 qq Thus we can begin to bound individual components of Γp𝑠0 q as ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋p𝑠𝑡 qq ď ℓp𝜋 1 p𝑠𝑡1 qqq ´ ℓp𝜋 1 p𝑠𝑡 qq ` 𝛽 rℓp𝜋p𝑠 ˆ 𝑡 qq ´ ℓp𝜋p𝑠𝑡 qqs Since ℓ is 𝐿-Lipschitz continuous, we have › › ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋 1 p𝑠𝑡 qq ď 𝐿 ›𝜋 1 p𝑠𝑡1 q ´ 𝜋 1 p𝑠𝑡 q› › › ď 𝐿𝛾 ›𝑠1 ´ 𝑠𝑡 › 𝑡 (C.4) where (C.4) is due to the smoothness condition [2] of policy class Π. Given a policy class Π with 𝛾 ă 1, the following claim can be proved by induction: Claim: }𝑠𝑡1 ´ 𝑠𝑡 } ď p1´ 𝛽𝛽𝜖 qp1´𝛾 q Proof. For the base case, given the same starting state 𝑠0 , we have 𝑠11 “ r𝑥 1 , 𝜋 1 p𝑠0 qs and 𝑠1 “ r𝑥1 , 𝜋p𝑠0 qs. Thus › 1 › › › ›𝑠 ´ 𝑠1 › “ ›𝜋 1 p𝑠0 q ´ 𝜋p𝑠0 q› “ }𝛽 𝜋p𝑠 ˆ 0 q ` p1 ´ 𝛽q𝜋p𝑠0 q ´ 𝜋p𝑠0 q} 1 “ 𝛽 }𝜋p𝑠 ˆ 0 q ´ 𝜋p𝑠0 q} ď 𝛽𝜖 ď 𝛽𝜖 p1 ´ 𝛽qp1 ´ 𝛾q › › › › In the inductive case, assume we have ›𝑠𝑡1´1 ´ 𝑠𝑡 ´1 › ď p1´ 𝛽𝛽𝜖 . Then similar to qp1´𝛾 q 221 before, the definition of 𝑠𝑡1 and 𝑠𝑡 leads to › 1 › ›“ › ‰ ›𝑠 ´ 𝑠𝑡 › “ › 𝑥𝑡 , 𝜋 1 p𝑠1 q ´ r𝑥𝑡 , 𝜋p𝑠𝑡 ´1 qs› 𝑡 𝑡 ´1 › 1 1 › › “ 𝜋 p𝑠𝑡 ´1 q ´ 𝜋p𝑠𝑡 ´1 q› › › › › ď ›𝜋 1 p𝑠𝑡1´1 q ´ 𝜋 1 p𝑠𝑡 ´1 q› ` ›𝜋 1 p𝑠𝑡 ´1 q ´ 𝜋p𝑠𝑡 ´1 q› › › ď 𝛾 ›𝑠𝑡1´1 ´ 𝑠𝑡 ´1 › ` 𝛽 }𝜋p𝑠 ˆ 𝑡 ´1 q ´ 𝜋p𝑠𝑡 ´1 q} 𝛽𝜖 ` 𝛽𝜖 ď𝛾 p1 ´ 𝛽qp1 ´ 𝛾q 𝛽𝜖 ď p1 ´ 𝛽qp1 ´ 𝛾q Applying the claim to equation (C.4), we have ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋 1 p𝑠𝑡 qq ď 𝛽𝛾𝜖 𝐿 p1 ´ 𝛽qp1 ´ 𝛾q which leads to ℓp𝜋 1 p𝑠𝑡1 q ´ ℓp𝜋p𝑠𝑡 qqq ď 𝛽𝛾𝜖 𝐿 p1 ´ 𝛽qp1 ´ 𝛾q ` 𝛽pℓp𝜋p𝑠 ˆ 𝑡 qq ´ ℓp𝜋p𝑠𝑡 qqq (C.5) Integrating (C.5) over the starting state 𝑠0 „ 𝜇 and input trajectories t𝑥𝑡 u𝑇𝑡“1 , we arrive at the policy improvement bound: ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝛾𝜖 𝐿 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq p1 ´ 𝛽qp1 ´ 𝛾q where ℓ𝜋 p𝜋q ˆ is the expected loss of the trained policy 𝜋ˆ on the state distribution induced by policy 𝜋 (reduction term, analogous to policy advantage in the traditional MDP terminologies (Kakade and Langford, 2002)) This means in the worst case, as we choose 𝛽 Ñ 0, we have rℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋qs Ñ 0, meaning the new policy does not degrade much for a small choice of 𝛽. However if ℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋q ! 0, we can choose 𝛽 to enforce monotonic improvement of the policy by adaptively choosing 𝛽 that minimizes the right-hand side. In particular, let ´𝛿 the reduction term be Δ “ ℓ𝜋 p𝜋q ´ ℓ𝜋 p𝜋q ˆ ą 0 and let 𝛿 “ 1𝛾𝜖´𝐿𝛾 , then for 𝛽 “ Δ2Δ we have the following monotonic policy improvement: ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď ´ pΔ ´ 𝛿q2 2pΔ ` 𝛿q 222 Proof of theorem 4.5.5 - 𝑇-dependent improvement Theorem Statement. (theorem 4.5.5) Assume ℓ is convex and 𝐿-Lipschitz, and Condition 1 holds. Let 𝜖 “ max }𝜋p𝑠q ˆ ´ 𝜋p𝑠q}. Then: 𝑠„𝑑 𝜋 ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝜖 𝐿𝑇 ` 𝛽 pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq . In particular, choosing 𝛽 P p0, 1{𝑇q yields: ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝜖 𝐿 ` 𝛽 pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq . Proof. The proof of theorem 4.5.5 largely follows the structute of theorem 4.5.6, except that we are using the slighty weaker Condition 1 which leads to weaker bound on the policy improvement that depends on the trajectory horizon 𝑇. For any state 𝑠0 taken from the starting state distribution 𝜇, sequentially roll-out policies 𝜋 1 and 𝜋 to receive two separate trajectories 𝜋 1 : 𝑠0 Ñ 𝑠11 Ñ . . . Ñ 𝑠𝑇1 and 𝜋 1 : 𝑠0 Ñ 𝑠1 Ñ . . . Ñ 𝑠𝑇 . Consider a pair of states 𝑠𝑡1 “ r𝑥𝑡 , 𝜋 1 p𝑠𝑡1´1 qs and 𝑠𝑡 “ r𝑥𝑡 , 𝜋p𝑠𝑡 ´1 qs corresponding to the same input feature 𝑥𝑡 , as before we can decompose ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋p𝑠𝑡 qq “ ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋 1 p𝑠𝑡 qq ` ℓp𝜋 1 p𝑠𝑡 qq ´ ℓp𝜋p𝑠𝑡 qq ď 𝐿 }𝜋 1 p𝑠𝑡1 q ´ 𝜋 1 p𝑠𝑡 q} ` 𝛽pℓp𝜋p𝑠 ˆ 𝑡 qq ´ ℓp𝜋p𝑠𝑡 qqq due to convexity and 𝐿-Lipschitz continuity of ℓ. Condition 1 further yields: ℓp𝜋 1 p𝑠𝑡1 qq ´ ℓp𝜋p𝑠𝑡 qq ď 𝐿 }𝑠𝑡1 ´ 𝑠𝑡 } ` 𝛽pℓp𝜋p𝑠 ˆ 𝑡 qq ´ ℓp𝜋p𝑠𝑡 qqq. By the construction of the states, note that › 1 › › › ›𝑠 ´ 𝑠𝑡 › “ ›𝜋 1 p𝑠1 q ´ 𝜋p𝑠𝑡 ´1 q› 𝑡 𝑡 ´1 › 1 1 › › › › ď 𝜋 p𝑠𝑡 ´1 q ´ 𝜋 1 p𝑠𝑡 ´1 q› ` ›𝜋 1 p𝑠𝑡 ´1 q ´ 𝜋p𝑠𝑡 ´1 q› › › ď ›𝑠𝑡1´1 ´ 𝑠𝑡 ´1 › ` 𝛽p}𝜋p𝑠 ˆ 𝑡 ´1 q ´ 𝜋p𝑠𝑡 ´1 q}q › 1 › ď ›𝑠𝑡 ´1 ´ 𝑠𝑡 ´1 › ` 𝛽𝜖 (by condition 1 and definition of 𝜖). From here, one can use this recursive relation to easily show that }𝑠𝑡1 ´ 𝑠𝑡 } ď 𝛽𝜖𝑡 for all 𝑡 P r1, 𝑇s. Averaging over the 𝑇 time steps and integrating over the starting state distribution, we have: ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝛽𝜖 𝐿p𝑇 ` 1q{2 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq ď 𝛽𝜖 𝐿𝑇 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq In particular, 𝛽 P p0, 1{𝑇q yields ℓ𝜋1 p𝜋 1 q ´ ℓ𝜋 p𝜋q ď 𝜖 𝐿 ` 𝛽pℓ𝜋 p𝜋q ˆ ´ ℓ𝜋 p𝜋qq. 223 Proof of proposition 4.5.8 - smooth expert proposition Proposition Statement. (Proposition 4.5.8) Let 𝜔 be the average supervised training error from F , i.e. 𝜔 “ min E𝑥 „X r} 𝑓 pr𝑥, 0sq ´ 𝑎 ˚ }s. Let the rolled-out trajectory 𝑓 PF of current policy 𝜋 be t𝑎 𝑡 u. If the average gap between 𝜋 and 𝜋 ˚ is such that E𝑡 „Uniformr1:𝑇 s r}𝑎 𝑡˚ ´ 𝑎 𝑡 ´1 }s ě 3𝜔 ` 𝜂p1 ` 𝜆q, then using t𝑎 𝑡˚ u as feedback will cause the trained policy 𝜋ˆ to be non-smooth, i.e.: E𝑡 „Uniformr1:𝑇 s r}𝑎ˆ 𝑡 ´ 𝑎ˆ 𝑡 ´1 }s ě 𝜂, for t𝑎ˆ 𝑡 u the rolled-out trajectory of 𝜋. ˆ Proof. Recall that Π𝜆 is formed by regularizing a class of supervised learners F with the singleton class of smooth function H fi tℎp𝑎q “ 𝑎u, via a hyper-parameter 𝜆 that controls the trade-off between being close to the two classes. Minimizing over Π𝜆 can be seen as a regularized optimization problem: 𝜋p𝑥, ˆ 𝑎q “ arg min ℓp𝜋pr𝑥, 𝑎sqq 𝜋 PΠ “ arg min p 𝑓 p𝑥, 𝑎q ´ 𝑎 ˚ q2 ` 𝜆p 𝑓 p𝑥, 𝑎q ´ ℎp𝑎qq2 𝑓 PF ,ℎPH “ arg minp 𝑓 p𝑥, 𝑎q ´ 𝑎 ˚ q2 ` 𝜆p 𝑓 p𝑥, 𝑎q ´ 𝑎q2 (C.6) 𝑓 PF where hyper-parameter 𝜆 trades-off the distance of 𝑓 p𝑥, 𝑎q relative to 𝑎 (smoothness) and 𝑎 ˚ (imitation accuracy), and 𝑎 P R1 . Such a policy 𝜋, at execution time, corresponds to the regularized minimizer of: 𝑎 𝑡 “ 𝜋pr𝑥, 𝑎 𝑡 ´1 sq “ arg min }𝑎 ´ 𝑓 pr𝑥𝑡 , 𝑎 𝑡 ´1 sq}2 ` 𝜆 }𝑎 ´ 𝑎 𝑡 ´1 }2 𝑎 𝑓 pr𝑥𝑡 , 𝑎 𝑡 ´1 sq ` 𝜆𝑎 𝑡 ´1 “ 1`𝜆 (C.7) where 𝑓 P F is the minimizer of equation C.6 Thus we enforce smoothness of learning policy from Π𝜆 by encouraging low first order difference of consecutive actions of the executed trajectory t𝑎 𝑡 u. In practice, we may contrain this first order difference relative › › to the human trajectory ř ř › 𝑇 𝑇 1 1 ˚ ˚ › 𝑡 “1 }𝑎 𝑡 ´ 𝑎 𝑡 ´1 } ď 𝜂, where 𝜂9 𝑇 𝑡 “1 ›𝑎 𝑡 ´ 𝑎 𝑡 ´1 ›. 𝑇 Consider any given iteration with the following set-up: we execute old policy 𝜋 “ 𝜋 𝑜𝑙𝑑 to get rolled-out trajectory t𝑎 𝑡 u𝑇𝑡“1 . Form the new data set as D “ tp𝑠𝑡 , 𝑎 𝑡˚ qu𝑇𝑡“1 224 with predictors 𝑠𝑡 “ r𝑥𝑡 , 𝑎 𝑡 ´1 s and feedback labels simply the human actions 𝑎 𝑡˚ . Use this data set to train a policy 𝜋ˆ by learning a supervised 𝑓ˆ P F from D. Similar to 𝜋, the execution of 𝜋ˆ corresponds to 𝑎ˆ 𝑡 where: 𝑎ˆ 𝑡 “ 𝜋pr𝑥 ˆ 𝑡 , 𝑎ˆ 𝑡 ´1 sq ›2 › “ arg min ›𝑎 ´ 𝑓ˆpr𝑥𝑡 , 𝑎ˆ 𝑡 ´1 sq› ` 𝜆 }𝑎 ´ 𝑎ˆ 𝑡 ´1 }2 𝑎 𝑓ˆpr𝑥𝑡 , 𝑎ˆ 𝑡 ´1 sq ` 𝜆 𝑎ˆ 𝑡 ´1 “ 1`𝜆 (C.8) Denote by 𝑓0 the "naive" supervised learner from F . In other words, 𝑓0 “ 𝑇 ř } 𝑓 pr𝑥𝑡 , 0sq ´ 𝑎 𝑡˚ }2 . Let 𝜔 be the average gap between human trajecarg min 𝑓 PF 𝑡 “1 tory and the rolled-out trajectory of 𝑓0 , i.e. 𝑇 𝜔“ 1ÿ } 𝑓0 pr𝑥𝑡 , 0sq ´ 𝑎 𝑡˚ } 𝑇 𝑡 “1 Note that it is reasonable to assume that the average errors of 𝑓 and 𝑓ˆ are no worse than 𝑓0 , since in the worst case we can simply discard the extra features 𝑎 𝑡 ´1 (resp. 𝑎ˆ 𝑡 ´1 ) of 𝑓 (resp. 𝑓ˆ) to recover the performance of the naive learner 𝑓0 : 𝑇 1ÿ } 𝑓 pr𝑥𝑡 , 𝑎 𝑡 ´1 sq ´ 𝑎 𝑡˚ } ď 𝜔 𝑇 𝑡 “1 𝑇 › 1 ÿ ›› ˆ 𝑓 pr𝑥𝑡 , 𝑎ˆ 𝑡 ´1 sq ´ 𝑎 𝑡˚ › ď 𝜔 𝑇 𝑡 “1 Assume that the old policy 𝜋 “ 𝜋 𝑜𝑙𝑑 is "bad" in the sense that the rolled-out trajectory t𝑎 𝑡 u𝑇𝑡“1 differs substantially from human trajectory t𝑎 𝑡˚ u𝑇𝑡“1 . Specifically, denote the gap: 𝑇 1ÿ ˚ }𝑎 ´ 𝑎 𝑡 ´1 } “ Ω " 𝜔 𝑇 𝑡 “1 𝑡 This means the feedback correction 𝑎 𝑡˚ to 𝑠𝑡 “ r𝑥𝑡 , 𝑎 𝑡 ´1 s is not smooth. We will show that the trained policy 𝜋ˆ from D will not be smooth. From the definition of 𝑎 𝑡 and 𝑎ˆ 𝑡 from equations C.7 and C.8, we have for each 𝑡: 𝑓 pr𝑥𝑡 , 𝑎 𝑡 ´1 sq ´ 𝑓ˆpr𝑥𝑡 , 𝑎ˆ 𝑡 ´1 sq 𝜆 p𝑎 𝑡 ´1 ´ 𝑎ˆ 𝑡 ´1 q ` 1`𝜆 1`𝜆 Applying triangle inequality and summing up over 𝑡, we have: 𝑎 𝑡 ´ 𝑎ˆ 𝑡 “ 𝑇 1ÿ }𝑎 𝑡 ´ 𝑎ˆ 𝑡 } ď 2𝜔 𝑇 𝑡 “1 225 From here we can provide a lower bound on the smoothness of the new trajectory ř 𝑎ˆ 𝑡 , as defined by the first order difference 𝑇1 𝑇𝑡“1 }𝑎ˆ 𝑡 ´ 𝑎ˆ 𝑡 ´1 }. By definition of 𝑎ˆ 𝑡 : › › › 𝑓ˆpr𝑥 , 𝑎ˆ sq ´ 𝑎ˆ › › 𝑡 𝑡 ´1 𝑡 ´1 › }𝑎ˆ 𝑡 ´ 𝑎ˆ 𝑡 ´1 } “ › › › › 1`𝜆 › › › 𝑓ˆpr𝑥 , 𝑎ˆ sq ´ 𝑎 ˚ ` 𝑎 ˚ ´ 𝑎 › ` 𝑎 ´ 𝑎 ˆ 𝑡 𝑡 ´1 𝑡 ´1 𝑡 ´1 𝑡 ´1 › › 𝑡 𝑡 “› › › › 1`𝜆 › › }𝑎 𝑡˚ ´ 𝑎 𝑡 ´1 } ´ › 𝑓ˆpr𝑥𝑡 , 𝑎ˆ 𝑡 ´1 sq ´ 𝑎 𝑡˚ › ´ }𝑎 𝑡 ´1 ´ 𝑎ˆ 𝑡 ´1 } ě 1`𝜆 Again summing up over 𝑡 and taking the average, we obtain: 𝑇 1ÿ Ω ´ 3𝜔 }𝑎ˆ 𝑡 ´ 𝑎ˆ 𝑡 ´1 } ě 𝑇 𝑡 “1 1`𝜆 Hence for Ω " 𝜔, meaning the old trajectory is sufficiently far away from the ideal human trajectory, setting the learning target to be the ideal human actions will cause the learned trajectory to be non-smooth. C.2 Imitation Learning With Smooth Regression Forests Variant of SIMILE Using Smooth Regression Forest Policy Class We provide a specific instantiation of algorithm 4 that we used for our experiment, based on a policy class Π as a smooth regularized version of the space of tree-based ensembles. In particular, F is the space of random forests and H is the space of ř linear auto-regressors H fi tℎp𝑎 𝑡 ´1:𝑡 ´𝜏 q “ 𝑖𝜏“1 𝑐𝑖 𝑎 𝑡 ´𝑖 u. In combination, F and H form a complex tree-based predictor that can predict smooth sequential actions. Empirically, decision tree-based ensembles are among the best performing supervised machine learning method (Caruana and Niculescu-Mizil, 2006; Criminisi et al., 2012). Due to the piece-wise constant nature of decision tree-based prediction, the results are inevitably non-smooth. We propose a recurrent extension based on H , where the prediction at the leaf node is not necessarily a constant, but rather is a smooth function of both static leaf node prediction and its previous predictions. By merging the powerful tree-based policy class with a linear auto-regressor, we provide a novel approach to train complex models that can accommodate smooth sequential prediction using model-based smooth regularizer, at the same time leveraging the expressiveness of complex model-free function class (one can similarly apply the framework to the space of neural networks). Algorithm 19, which is based on SIMILE, describes in more details our training procedure used for the automated 226 Algorithm 19 Imitation Learning for Online Sequence Prediction with Smooth Regression Forest Input: Input features X “ t𝑥𝑡 u𝑇𝑡“1 , expert demonstration A˚ “ t𝑎 𝑡˚ u𝑇𝑡“1 , base routine Forest, past horizon ” 𝜏, sequence ofı 𝜎 P p0, 1q ˚ 1: Initialize A0 Ð A , S0 Ð t 𝑥 𝑡:𝑡 ´𝜏 , 𝑎 𝑡˚´1:𝑡 ´𝜏 u, 𝑇 ` ˘2 ř ř ℎ0 “ arg min 𝑎 𝑡˚ ´ 𝑖𝜏“1 𝑐𝑖 𝑎 𝑡˚´𝑖 𝑐 1 ,...,𝑐 𝜏 𝑡 “1 2: Initial policy 𝜋0 “ 𝜋ˆ 0 ÐForestpS0 , A0 | ℎ0 q 3: for 𝑛 “ 1, . . . , 𝑁 do ” ı 4: 1 A𝑛 “ t𝑎 𝑡𝑛 u Ð t𝜋𝑛´1 p 𝑥𝑡:𝑡 ´𝜏 , 𝑎 𝑡𝑛´´1:𝑡 ´𝜏 qu 5: ı ” 𝑛 𝑛 S𝑛 Ð t𝑠𝑡 “ 𝑥𝑡:𝑡 ´𝜏 , 𝑎 𝑡 ´1:𝑡 ´𝜏 u 6: p 𝑛 “ tp A 𝑎 𝑡𝑛 “ 𝜎𝑎 𝑡𝑛 ` p1 ´ 𝜎q𝑎 𝑡˚ u @𝑠𝑡𝑛 P S𝑛 7: 𝑇 ` ˘2 ř ř ℎ𝑛 “ arg min 𝑎ˆ 𝑡𝑛 ´ 𝑖𝜏“1 𝑐𝑖 𝑎ˆ 𝑡𝑛´𝑖 //sequential roll-out old policy //Form states in 1d case // collect smooth 1-step feedback //update 𝑐𝑖 𝑐 1 ,...,𝑐 𝜏 𝑡 “1 8: p 𝑛 | ℎ𝑛 q 𝜋ˆ 𝑛 ÐForestpS𝑛 , A 9: 𝛽 Ð errorp𝜋ˆ q`errorp𝜋q via regularized least square // train with smooth decision forests. See section C.2 //set 𝛽 to weighted empirical errors // update policy errorp𝜋 q 10: 𝜋𝑛 “ 𝛽 𝜋ˆ 𝑛 ` p1 ´ 𝛽q𝜋𝑛´1 11: end for output Last policy 𝜋 𝑁 camera planning experiment. We first describe the role of the linear autoregressor class H , before discussing how to incorporate H into decision tree training to make smooth prediction (see the next section). The autoregresor ℎ 𝜋 p𝑎 ´1 , . . . , 𝑎 ´𝜏 q is typically selected from a class of autoregressors H . In our experiments, we use regularized linear autoregressors as H . Consider a generic learning policy 𝜋 with a rolled-out trajectory A “ t𝑎 𝑡 u𝑇𝑡“1 corresponding to the input sequence X “ t𝑥𝑡 u𝑇𝑡“1 . We form the state sequence S “ t𝑠𝑡 u𝑇𝑡“1 “ tr𝑥𝑡 , . . . , 𝑥𝑡 ´𝜏 , 𝑎 𝑡 ´1 , . . . , 𝑎 𝑡 ´𝜏 su𝑇𝑡“1 , where 𝜏 indicates the past horizon that is adequate to approximately capture the full state information. We approximate the smoothness of the trajectory A by a linear autoregressor 𝜏 ÿ ℎ 𝜋 ” ℎ 𝜋 p𝑠𝑡 q ” 𝑐𝑖 𝑎 𝑡 ´𝑖 𝑖 “1 for a (learned) set of coefficients t𝑐𝑖 u𝑖𝜏“1 such that 𝑎 𝑡 « ℎ 𝜋 p𝑠𝑡 q. Given feedback 227 p “ t𝑎ˆ 𝑡 u, the joint loss function thus becomes target A ℓp𝑎, 𝑎ˆ 𝑡 q “ ℓ𝑑 p𝑎, 𝑎ˆ 𝑡 q ` 𝜆ℓ𝑅 p𝑎, 𝑠𝑡 q 𝜏 ÿ 2 “ p𝑎 ´ 𝑎ˆ 𝑡 q ` 𝜆p𝑎 ´ 𝑐𝑖 𝑎 𝑡 ´𝑖 q2 𝑖 “1 Here 𝜆 trades off between smoothness versus absolute imitation accuracy. The autoregressor ℎ 𝜋 acts as a smooth linear regularizer, the parameters of which can be p according to updated at each iteration based on feedback target A › ›2 ›p › p ℎ 𝜋 “ arg min ›A ´ ℎpAq› ℎ PH 𝑇 ÿ “ arg minp 𝑐 1 ,...,𝑐 𝜏 𝑡 “1 𝜏 ÿ p𝑎ˆ 𝑡 ´ 𝑐𝑖 𝑎ˆ 𝑡 ´𝑖 q2 q, (C.9) 𝑖 “1 In practice we use a regularized version of equation (C.9) to learn a new set of coefficients t𝑐𝑖 u𝑖𝜏“1 . The Forest procedure (Line 8 of algorithm 2) would use this updated ℎ 𝜋 to train a new policy that optimizes the trade-off between 𝑎 𝑡 « 𝑎ˆ 𝑡 ř (feedback) versus smoothness as dictated by 𝑎 𝑡 « 𝑖𝜏“1 𝑐𝑖 𝑎 𝑡 ´𝑖 . Smooth Regularization with Linear Autoregressors Our application of Algorithm 1 to realtime camera planning proceeds as follows: At each iteration, we form a state sequence S based on the rolled-out trajectory A and tracking input data X such that 𝑠𝑡 “ r𝑥𝑡 , . . . , 𝑥𝑡 ´𝜏 , 𝑎 𝑡 ´1 , . . . , 𝑎 𝑡 ´𝜏 s for appropriate 𝜏 p that captures the history of the sequential decisions. We generate feedback targets A based on each 𝑠𝑡 P S following 𝑎ˆ 𝑡 “ 𝜎𝑎 𝑡 ` p1 ´ 𝜎q𝑎 𝑡˚ using a parameter 𝜎 P p0, 1q depending on the Euclidean distance between A and A˚ . Typically, 𝜎 gradually decreases to 0 as the rolled-out trajectory improves on the training set. After generating the targets, a new linear autoregressor ℎ 𝜋 (new set of coefficients t𝑐𝑖 u𝑖𝜏“1 ) p using regularized least squares (as described in the previous is learned based on A p and the updated coefficients section). We then train a new model 𝜋ˆ based on S, A, t𝑐𝑖 u, using Forest - our recurrent decision tree framework that is capable of generating smooth predictions using autoregressor ℎ 𝜋 as a smooth regularizer (see the following section for how to train smooth decision trees). Note that typically this creates a "chicken-and-egg" problem. As the newly learned policy 𝜋ˆ is greedily p the rolled-out trajectory of 𝜋ˆ may have a state distribution trained with respect to A, that is different from what the previously learned ℎ 𝜋 would predict. Our approach offers two remedies to this circular problem. First, by allowing feedback signals 228 to vary smoothly relative to the current rolled-out trajectory A, the new policy 𝜋ˆ should induce a new autoregresor that is similar to previously learned ℎ 𝜋 . Second, p eventually by interpolating distributions (Line 10 of Algorithm 2) and having A converge to the original human trajectory A˚ , we will have a stable and converging state distribution, leading to a stable and converging ℎ 𝜋 . Throughout iterations, the linear autoregressor ℎ 𝜋 and regularization parameter 𝜆 enforces smoothness of the rolled-out trajectory, while the recurrent decision tree framework Forest learns increasingly accurate imitation policy. We generally achieve a satisfactory policy after 5-10 iterations in our sport broadcasting data sets. In the following section, we describe the mechanics of our recurrent decision tree training. Smooth Regression Tree Training Given states 𝑠 as input, a decision tree specifies a partitioning of the input state space. Let 𝐷 “ tp𝑠𝑚 , 𝑎ˆ 𝑚 qu𝑚𝑀“1 denote a training set of state/target pairs. Conventional regression tree learning aims to learn a partitioning such that each leaf node, node, makes a constant prediction via minimizing the squared loss function: ÿ 𝑎¯ node “ arg min 𝑎 ℓ𝑑 p𝑎, 𝑎q ˆ p𝑠,𝑎ˆ qP 𝐷 node ÿ “ arg min 𝑎 p𝑎ˆ ´ 𝑎q2 , (C.10) p𝑠,𝑎ˆ qP 𝐷 node where 𝐷 node denotes the training data from 𝐷 that has partitioned into the leaf node. For squared loss, we have: 𝑎¯ node “ mean t𝑎ˆ |p𝑠, 𝑎q ˆ P 𝐷 node u . (C.11) In the recurrent extension to Forest, we allow the decision tree to branch on the input state 𝑠, which includes the previous predictions 𝑎 ´1 , . . . , 𝑎 ´𝜏 . To enforce more explicit smoothness requirements, let ℎ 𝜋 p𝑎 ´1 , . . . , 𝑎 ´𝜏 q denote an autoregressor that captures the temporal dynamics of 𝜋 over the distribution of input sequences 𝑑x , while ignoring the inputs 𝑥. At time step 𝑡, ℎ 𝜋 predicts the behavior 𝑎 𝑡 “ 𝜋p𝑠𝑡 q given only 𝑎 𝑡 ´1 , . . . , 𝑎 𝑡 ´𝜏 . Our policy class Π of recurrent decision trees 𝜋 makes smoothed predictions by regularizing the predictions to be close to its autoregressor ℎ 𝜋 . The new loss function incorporates both the squared distance loss ℓ𝑑 , as well as a smooth regularization 229 loss such that: ÿ L 𝐷 p𝑎q “ ℓ𝑑 p𝑎, 𝑎q ˆ ` 𝜆ℓ𝑅 p𝑎, 𝑠q p𝑠,𝑎ˆ qP 𝐷 ÿ “ p𝑎 ´ 𝑎q ˆ 2 ` 𝜆p𝑦 ´ ℎ 𝜋 p𝑠qq2 p𝑠,𝑎ˆ qP 𝐷 where 𝜆 is a hyper-parameter that controls how much we care about smoothness versus absolute distance loss. Making prediction: For any any tree/policy 𝜋, each leaf node is associated with the terminal leaf node value 𝑎¯ node such that prediction 𝑎˜ given input state 𝑠 is: 𝑎p𝑠q ˜ ” 𝜋p𝑠q “ arg min p𝑎 ´ 𝑎¯ nodep𝑠q q2 ` 𝜆p𝑎 ´ ℎ 𝜋 p𝑠qq2 , (C.12) 𝑎 “ 𝑎¯ nodep𝑠q ` 𝜆ℎ 𝜋 p𝑠q (C.13) . 1`𝜆 where nodep𝑠q denotes the leaf node of the decision tree that 𝑠 branches to. Setting terminal node value: Given a fixed ℎ 𝜋 and decision tree structure, navigating through consecutive binary queries eventually yields a terminal leaf node with associated training data 𝐷 node Ă 𝐷. One option is to set the terminal node value 𝑎¯ node to satisfy: ÿ ℓ𝑑 p𝑎p𝑠|𝑎q, ˜ 𝑎q ˆ 𝑎¯ node “ arg min 𝑎 p𝑠,𝑎ˆ qP 𝐷 node ÿ “ arg min 𝑎 p𝑎p𝑠|𝑎q ˜ ´ 𝑎q ˆ 2 p𝑠,𝑎ˆ qP 𝐷 node ˆ ÿ “ arg min 𝑎 (C.14) p𝑠,𝑎ˆ qP 𝐷 node 𝑎 ` 𝜆ℎ 𝜋 p𝑠q ´ 𝑎ˆ 1`𝜆 ˙2 for 𝑎p𝑠|𝑎q ˜ defined as in (C.13) with 𝑎 ” 𝑎¯ nodep𝑠q . Similar to (C.11), we can write the closed-form solution of (C.14) as: 𝑎¯ node “ mean tp1 ` 𝜆q𝑎ˆ ´ 𝜆ℎ 𝜋 p𝑠q |p𝑠, 𝑎q ˆ P 𝐷 node u . (C.15) When 𝜆 “ 0, (C.15) reduces to (C.11). Note that (C.14) only looks at imitation loss ℓ𝑑 , but not smoothness loss ℓ𝑅 . Alternatively in the case of joint imitation and smoothness loss, the terminal leaf node is set to minimize the joint loss function: 230 𝑎¯ node “ arg min L 𝐷 node p𝑎p𝑠|𝑎qq ˜ 𝑎 ÿ “ arg min ℓ𝑑 p𝑎p𝑠|𝑎q, ˜ 𝑎q ˆ ` 𝜆ℓ𝑅 p𝑎p𝑠|𝑎q, ˜ 𝑠q 𝑎 p𝑠,𝑎ˆ qP 𝐷 node ÿ p𝑎p𝑠|𝑎q ˜ ´ 𝑎q ˆ 2 ` 𝜆p𝑎p𝑠|𝑎q ˜ ´ ℎ 𝜋 p𝑠qq2 “ arg min 𝑎 (C.16) p𝑠,𝑎ˆ qP 𝐷 node ˆ 𝑎 ` 𝜆ℎ 𝜋 p𝑠q ´ 𝑎ˆ “ arg min 1`𝜆 𝑎 p𝑠,𝑎ˆ qP 𝐷 node ˆ ˙2 𝑎 ` 𝜆ℎ 𝜋 p𝑠q `𝜆 ´ ℎ 𝜋 p𝑠q 1`𝜆 ÿ ˙2 “ mean t𝑎ˆ |p𝑠, 𝑎q ˆ P 𝐷 node u , (C.17) Node splitting mechanism: For a node representing a subset 𝐷 node of the training data, the node impurity is defined as: 𝐼node “ L 𝐷 node p𝑎¯ node q ÿ “ ℓ𝑑 p𝑎¯ node , 𝑎q ˆ ` 𝜆ℓ𝑅 p𝑎¯ node , 𝑠q p𝑠,𝑎ˆ qP 𝐷 node ÿ “ p𝑎¯ node ´ 𝑎q ˆ 2 ` 𝜆p𝑎¯ node ´ ℎ 𝜋 p𝑠qq2 p𝑠,𝑎ˆ qP 𝐷 node where 𝑎¯ node is set according to equation (C.15) or (C.17) over p𝑠, 𝑎q’s ˆ in 𝐷 node . At each possible splitting point where 𝐷 node is partitioned into 𝐷 left and 𝐷 right , the impurity of the left and right child of the node is defined similarly. As with normal decision trees, the best splitting point is chosen as one that maximizes the impurity |𝐷 | |𝐷 left | reduction: 𝐼node ´ |𝐷 𝐼left ´ |𝐷right 𝐼right node | node | 231 Appendix D APPENDIX TO CHAPTER 5 D.1 Theoretical Analysis Preliminaries and Notations We formally define an ambient control policy space U to be a vector space equipped a with inner product x¨, ¨y : U ˆ U ÞÑ R, which induces a norm }𝑢} “ x𝑢, 𝑢y, and its dual norm defined as }𝑣}˚ “ suptx𝑣, 𝑢y| }𝑢} ď 1u. While multiple ways to define the inner product exist, for concreteness we can think of the example of ş square-integrable stationary policies with x𝑢, 𝑣y “ S 𝑢p𝑠q𝑣p𝑠q𝑑𝑠. The addition operator ` between two policies 𝑢, 𝑣 P U is defined as p𝑢 ` 𝑣qp𝑠q “ 𝑢p𝑠q ` 𝑣p𝑠q for all state 𝑠 P S. Scaling 𝜆𝑢 ` 𝜅𝑣 is defined similarly for scalar 𝜆, 𝜅. ş8 The cost functional of a control policy 𝑢 is defined as 𝐽p𝑢q “ 0 𝑐p𝑠p𝜏q, 𝑢p𝜏qq𝑑𝜏, ş or 𝐽p𝑢q “ S 𝑐p𝑠, 𝑢p𝑠qq𝑑𝜇𝑢 p𝑠q, where 𝜇𝑢 is the distribution of states induced by policy 𝑢. This latter example is equivalent to the standard notion of value function in reinforcement learning. Separate from the parametric representation issues, both programmatic policy class Π and neural policy class F , and by extension - the joint policy class H , are considered to live in the ambient vector space U. We thus have a common and well-defined notion of distance between policies from different classes. We make an important distinction between differentiability of 𝐽pℎq in the ambient policy space (non-parametric), versus differentiability in parameterization (parametric). For example, if Π is a class of decision-tree based policy, policies in Π may not be differentiable under representation. However, policies 𝜋 P Π might still be differentiable when considered as points in the ambient vector space U. We will use the following standard notion of gradient and differentiability from functional analysis: Definition D.1.1 (Subgradients). The subgradient of 𝐽 at ℎ, denoted B𝐽pℎq, is the non-empty set t𝑔 P H |@ 𝑗 P H : x 𝑗 ´ ℎ, 𝑔y ` 𝐽pℎq ď 𝐽p 𝑗qu Definition D.1.2 (Fréchet gradient). A bounded linear operator ∇ : H ÞÑ H is 𝐽 p ℎ`𝑔 q´𝐽 p ℎq´x∇𝐽 p ℎq,𝑔 y “0 called Fréchet functional gradient of 𝐽 at ℎ P H if lim }𝑔 } }𝑔}Ñ0 232 The notions of convexity, smoothness and Bregman divergence are analogous to finite-dimensional setting: Definition D.1.3 (Strong convexity). A differentiable function 𝑅 is 𝛼´strongly convex w.r.t norm }¨} if 𝑅p𝑦q ě 𝑅p𝑥q ` x∇𝑅p𝑥q, 𝑦 ´ 𝑥y ` 𝛼2 }𝑦 ´ 𝑥}2 Definition D.1.4 (Lipschitz continuous gradient smoothness). A differentiable function 𝑅 is 𝐿 𝑅 ´strongly smooth w.r.t norm }¨} if }∇𝑅p𝑥q ´ ∇𝑅p𝑦q}˚ ď 𝐿 𝑅 }𝑥 ´ 𝑦} Definition D.1.5 (Bregman Divergence). For a strongly convex regularizer 𝑅, 𝐷 𝑅 p𝑥, 𝑦q “ 𝑅p𝑥q ´ 𝑅p𝑦q ´ x∇𝑅p𝑦q, 𝑥 ´ 𝑦y is the Bregman divergence between 𝑥 and 𝑦 (not necessarily symmetric) The following standard result for Bregman divergence will be useful: Lemma D.1.1. (Beck and Teboulle, 2003) For all 𝑥, 𝑦, 𝑧 we have the identity x∇𝑅p𝑥q ´ ∇𝑅p𝑦q, 𝑥 ´ 𝑧y “ 𝐷 𝑅 p𝑥, 𝑦q ` 𝐷 𝑅 p𝑧, 𝑥q ´ 𝐷 𝑅 p𝑧, 𝑦q. Since Bregman divergence is non-negative, a consequence of this identity is that 𝐷 𝑅 p𝑧, 𝑥q ´ 𝐷 𝑅 p𝑧, 𝑦q ď x∇𝑅p𝑥q ´ ∇𝑅p𝑦q, 𝑧 ´ 𝑥y Expected Regret Bound under Noisy Policy Gradient Estimates and Projection Errors In this section, we show regret bound for the performance of the sequence of returned programs 𝜋1 , . . . , 𝜋𝑇 of the algorithm. The analysis here is agnostic to the particular implementation of algorithm 6 and algorithm 7. Let 𝑅 be a 𝛼´strongly convex and 𝐿 𝑅 ´smooth functional with respect to norm }¨} on H . The steps from algorithm 5 can be described as follows. • Initialize 𝜋0 P Π. For each iteration 𝑡: p 1. Obtain a noisy estimate of the gradient ∇𝐽p𝜋 𝑡 ´1 q « ∇𝐽p𝜋𝑡 ´1 q p 2. Update in the H space: ∇𝑅pℎ𝑡 q “ ∇𝑅p𝜋𝑡 ´1 q ´ 𝜂 ∇𝐽p𝜋 𝑡 ´1 q 3. Obtain approximate projection 𝜋𝑡 “ Project𝜋𝑅 pℎ𝑡 q « arg min 𝑝𝑖PΠ 𝐷 𝑅 p𝜋, ℎ𝑡 q This procedure is an approximate functional mirror descent scheme under bandit feedback. We will develop the following result, which is a more detailed version of 5.4.1 in the main paper. 233 In the statement below, 𝐷 is the diameter on Π with respect to defined norm }¨} (i.e., 𝐷 “ sup }𝜋 ´ 𝜋 1 }). 𝐿 𝐽 is the Lipschitz constant of the functional 𝐽 on H . 𝛽, 𝜎 2 are the bound on the bias and variance of the gradient estimate at each iteration, respectively. 𝛼 and 𝐿 𝑅 are the strongly convex and smooth coefficients of the functional regularizer 𝑅. Finally, 𝜖 is the bound on the projection error with respect to the same norm }¨}. Theorem D.1.2 (Regret bound of returned policies). Let 𝜋1 , . . . , 𝜋𝑇 be a sequence of programmatic policies returned by algorithm 5 and 𝜋 ˚ be the optimal programmatic policy. We have the expected regret bound: « ff 𝑇 1ÿ 𝐿 𝑅 𝐷 2 𝜖 𝐿 𝑅 𝐷 𝜂p𝜎 2 ` 𝐿 2𝐽 q ˚ E ` ` ` 𝛽𝐷 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 q ď 𝑇 𝑡 “1 𝜂𝑇 𝜂 𝛼 b1 In particular, choosing the learning rate 𝜂 “ into: 𝑇 `𝜖 𝜎2 , the expected regret is simplified (D.1) « ff ˜ c 1ÿ E 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝑇 𝑡 “1 𝜎 𝑇 ¸ 1 `𝜖 `𝛽 𝑇 s 𝑡 “ Er∇ p 𝑡 |𝜋𝑡 s be the conditional expectation of the Proof. At each round 𝑡, let ∇ gradient estimate. We will use the shorthand notation ∇𝑡 “ ∇𝐽p𝜋𝑡 q. Denote the › › upper-bound on the bias of the estimate by 𝛽𝑡 , i.e., ›s ∇𝑡 ´ ∇𝑡 ›˚ ď 𝛽𝑡 almost surely. ›2 ‰ “ ›› › 2 s p p s Denote the noise of the gradient estimate by 𝜉𝑡 “ ∇𝑡 ´ ∇𝑡 , and 𝜎𝑡 “ E ›∇𝑡 ´ ∇𝑡 › ˚ p𝑡 . is the variance of gradient estimate ∇ › › ›𝜋𝑡 ´ Project 𝑅 p 𝑓𝑡 q› “ The projection operator is 𝜖´approximate in the sense that Π › › › { 𝑅 › ›ProjectΠ pℎ𝑡 q ´ ProjectΠ𝑅 pℎ𝑡 q› ď 𝜖 with some constant 𝜖, which reflects the statistical error of the imitation learning procedure. This projection error in general is independent of the choice of function classes Π and F .We will use the shorthand notation 𝜋𝑡˚ “ ProjectΠ𝑅 p 𝑓𝑡 q for the true Bregman projection of ℎ𝑡 onto Π. Due to convexity of 𝐽 over the space H (which includes Π), we have for all 𝜋 P Π: 𝐽p𝜋𝑡 q ´ 𝐽p𝜋q ď x∇𝑡 , 𝜋𝑡 ´ 𝜋y 234 We proceed to bound the RHS, starting with bounding the inner product where the actual gradient is replaced by the estimated gradient. p 𝑡 , 𝜋𝑡 ´ 𝜋y “ x∇ 1 x∇𝑅p𝜋𝑡 q ´ ∇𝑅pℎ𝑡 `1 q, 𝜋𝑡 ´ 𝜋y 𝜂𝑡 ˘ 1` 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, ℎ𝑡 `1 q ` 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q 𝜂𝑡 ˘ 1` ď 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡˚`1 q ´ 𝐷 𝑅 p𝜋𝑡˚`1 , ℎ𝑡 `1 q ` 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q 𝜂𝑡 1` “ 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ` 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡˚`1 q 𝜂𝑡 loooooooooooooomoooooooooooooon loooooooooooooooomoooooooooooooooon “ telescoping (D.2) (D.3) (D.4) projection error ˚ ´𝐷 𝑅 p𝜋𝑡 `1 , ℎ𝑡 `1 q ` 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q looooooooooooooooooomooooooooooooooooooon ˘ (D.5) relative improvement Equation (D.2) is due to the gradient update rule in F space. Equation (D.3) is derived from definition of Bregman divergence. Equation (D.4) is due to the generalized Pythagorean theorem of Bregman projection 𝐷 𝑅 p𝑥, 𝑦q ě 𝐷 𝑅 p𝑥, ProjectΠ𝑅 p𝑥qq` 𝐷 𝑅 pProjectΠ𝑅 p𝑥q, 𝑦q. The RHS of equation (D.4) are decomposed into three components that will be bounded separately. Bounding projection error. By lemma (D.1.1) we have 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡˚`1 q ď x∇𝑅p𝜋𝑡 `1 q ´ ∇𝑅p𝜋𝑡˚`1 q, 𝜋 ´ 𝜋𝑡 `1 y › › ď ›∇𝑅p𝜋𝑡 `1 q ´ ∇𝑅p𝜋𝑡˚`1 q› }𝜋 ´ 𝜋𝑡 `1 }˚ › › ď 𝐿 𝑅 ›𝜋𝑡 `1 ´ 𝜋𝑡˚`1 › 𝐷 ď 𝜖 𝐿 𝑅 𝐷 (D.6) (D.7) (D.8) Equation (D.7) is due to Cauchy–Schwarz. Equation (D.8) is due to Lipschitz smoothness of ∇𝑅 and definition of 𝜖´approximate projection. Bounding relative improvement. This follows standard argument from analysis of mirror descent algorithm. 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q ´ 𝐷 𝑅 p𝜋𝑡˚`1 , ℎ𝑡 `1 q “ 𝑅p𝜋𝑡 q ´ 𝑅p𝜋𝑡˚`1 q ` x∇𝑅pℎ𝑡 `1 q, 𝜋𝑡˚`1 ´ 𝜋𝑡 y (D.9) ›2 𝛼› ď x∇𝑅p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋𝑡˚`1 y ´ ›𝜋𝑡˚`1 ´ 𝜋𝑡 ›˚ ` x∇𝑅pℎ𝑡 `1 q, 𝜋𝑡˚`1 ´ 𝜋𝑡 y 2 ›2 𝛼 ›› ˚ ˚ p 𝜋 𝑡 `1 ´ 𝜋 𝑡 › “ ´𝜂𝑡 x∇𝑡 , 𝜋𝑡 `1 ´ 𝜋𝑡 y ´ 2 › › 2 2 2 𝜂 ›p › 𝜂𝑡 2 ď 𝑡 ›∇ p𝜎𝑡 ` 𝐿 2𝐽 q 𝑡› ď ˚ 2𝛼 𝛼 (D.10) (D.11) (D.12) 235 Equation (D.10) is from the 𝛼´strong convexity property of regularizer 𝑅. Equation (D.11) is by definition of the gradient update. Combining the bounds on the three components and taking expectation, we thus have ˆ ˙ ” ı 𝜂𝑡2 2 1 2 p E x∇𝑡 , 𝜋𝑡 ´ 𝜋y ď 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ` 𝜖 𝐿 𝑅 𝐷 ` p𝜎𝑡 ` 𝐿 𝐽 q 𝜂𝑡 𝛼 (D.13) p 𝑡 and actual gradient ∇𝑡 factors Next, the difference between estimated gradient ∇ into the bound via Cauchy-Schwarz: › ” ı › › › p p E x∇𝑡 ´ ∇𝑡 , 𝜋𝑡 ´ 𝜋y ď ›∇𝑡 ´ Er∇𝑡 s› }𝜋𝑡 ´ 𝜋} ď 𝛽𝑡 𝐷 (D.14) ˚ The results can be deduced from equations (D.13) and (D.14). Unbiased gradient estimates. For the case when the gradient estimate is unbiased, assume the variance of the noise of gradient estimates is bounded by 𝜎 2 , we have the expected regret bound for all 𝑝𝑖 P Π ff « 𝑇 𝐿 𝑅 𝐷 2 𝜖 𝐿 𝑅 𝐷 𝜂p𝜎 2 ` 𝐿 2𝐽 q 1ÿ 𝐽p𝜋𝑡 q ´ 𝐽p𝜋q ď ` ` (D.15) E 𝑇 𝑡 “1 𝜂𝑇 𝜂 𝛼 here to clarify, 𝐿 𝑅 is the smoothness coefficient of regularizer 𝑅 (i.e., the gradient of 𝑅 is 𝐿 𝑅 -Lipschitz, 𝐿 𝐽 is Lipschitz constant of 𝐽, 𝐷 is the diameter of Π under norm }¨}, 𝜎 2 is the upper-bound on the variance of gradient estimates, and 𝜖 is the error from the projection procedure (i.e., imitation learning loss). b1 `𝜖 We can set learning rate 𝜂 “ 𝑇𝜎2 to observe that the expected regret is bounded b by 𝑂p𝜎 𝑇1 ` 𝜖q. Biased gradient estimates. Assume that the bias of gradient estimate at each round is upper-bounded by 𝛽𝑡 ď 𝛽. Similar to before, combining inequalities from (D.13) and (D.14), we have « ff 𝑇 1ÿ 𝐿 𝑅 𝐷 2 𝜖 𝐿 𝑅 𝐷 𝜂p𝜎 2 ` 𝐿 2𝐽 q E 𝐽p𝜋𝑡 q ´ 𝐽p𝜋q ď ` ` ` 𝛽𝐷 (D.16) 𝑇 𝑡 “1 𝜂𝑇 𝜂 𝛼 b1 `𝜖 Similar to before, we can set learning rate 𝜂 “ 𝑇𝜎2 to observe that on the expected b regret is bounded by 𝑂p𝜎 𝑇1 ` 𝜖 ` 𝛽q. Compared to the bound on (D.15), in the biased case, the extra regret incurred per bound is simply a constant, and does not depend on 𝑇. 236 Finite-Sample Analysis In this section, we provide overall finite-sample analysis for Propel under some simplifying assumptions. We first consider the case where exact gradient estimate is available, before extending the result to the general case of noisy policy gradient update. Combining the two steps will give us the proof for the following statement (theorem 5.4.2 in the main paper) Theorem D.1.3 (Finite-sample guarantee). At each iteration, we perform vanilla policy gradient estimate of 𝜋 (over H ) using 𝑚 trajectories algorithm cand use DAgger b ` logp𝑇 {𝛿q ˘ 1 1 𝐻 to collect 𝑀 roll-outs. Setting the learning rate 𝜂 “ ` ` , 2 𝑇 𝑀 𝑀 𝜎 after 𝑇 rounds of the algorithm, we have that ¨ d ˛ c 𝑇 ÿ logp𝑇{𝛿q ‚ 1 1 𝐻 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď𝑂 ˝𝜎 ` ` 𝑇 𝑡 “1 𝑇 𝑀 𝑀 ˜ c ¸ logp𝑇 𝑘{𝛿q 𝐴𝐻 logp𝑇 𝑘{𝛿q `𝑂 𝜎 ` 𝑚 𝑚 holds with probability at least 1 ´ 𝛿, with 𝐻 the task horizon, 𝐴 the cardinality of action space, 𝜎 2 the variance of policy gradient estimates, and 𝑘 the dimension Π’s parameterization. Exact gradient estimate case. Assuming that the policy gradients can be calculated exactly, it is straight-forward to provide high-probability guarantee for the effect of the projection error. We start with the following result, adapted from (Ross et al., 2011b) for the case of projection error bound. In this version of DAgger, we assume that we only collect a single (state, expert action) pair from each trajectory roll-out. Result is similar, with tighter bound, when multiple data points are collected along the trajectory. Lemma D.1.4 (Projection error bound from imitation learning procedure). Using DAgger as the imitation learning sub-routine for our Project operator in algorithm 7, let 𝑀 be the number of trajectories rolled-out for learning, and 𝐻 be the horizon of the task. With probability at least 1 ´ 𝛿, we have c 2ℓ𝑚𝑎𝑥 logp1{𝛿qq 2ℓ p1 ` 𝐻q 𝑚𝑎𝑥 r ` 𝐷 𝑅 p𝜋, 𝜋 ˚ q ď 𝑂p1{𝑀q ` 𝑀 𝑀 where 𝜋 is the result of Project, 𝜋 ˚ is the true Bregman projection of ℎ onto Π, and ℓ𝑚𝑎𝑥 is the maximum value of the imitation learning loss function 𝐷 𝑅 p¨, ¨q 237 The bound in lemma D.1.4 is simpler than previous imitation learning results with cost information ((Ross and Bagnell, 2014; Ross et al., 2011b). The reason is that the goal of the Project operator is more modest. Since we only care about the distance between the empirical projection 𝜋 and the true projection 𝜋 ˚ , the loss objective in imitation learning is simplified (i.e., this is only a regret bound), and we can disregard how well policies in Π can imitate the expert ℎ, as well as the performance of 𝐽p𝜋q relative to the true cost from the environment 𝐽pℎq. A consequence of this lemma is that for the number of trajectories at each round of log 1{𝛿 imitation learning 𝑀 “ 𝑂p 𝜖 2 q`𝑂p 𝐻𝜖 q, we have 𝐷 𝑅 p𝜋𝑡 , 𝜋𝑡˚ q ď 𝜖 with probability at least 1 ´ 𝛿. Applying union bound across 𝑇 rounds of learning, we obtain the following guarantee (under no gradient estimation error) Proposition D.1.5 (Finite-sample Projection Error Bound). To simplify the presentation of the result, we consider 𝐿 𝑅 , 𝐷, 𝐿, 𝛼 to be known constants. Using DAgger log 𝑇 {𝛿 algorithm to collect 𝑀 “ 𝑂p 𝜖 2 q ` 𝑂p 𝐻𝜖 q roll-outs at each iteration, we have the following regret guarantee after 𝑇 rounds of our main algorithm: 𝑇 1ÿ 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď 𝑂 𝑇 𝑡 “1 ˆ 1 𝜖 ` `𝜂 𝜂𝑇 𝜂 ˙ c b 1 𝐻 𝑇 ` 𝑀 ` with probability at least 1 ´ 𝛿. Consequently, setting 𝜂 “ we have that ¨d ˛ c 𝑇 ÿ logp𝑇{𝛿q ‚ 1 𝐻 1 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď 𝑂 ˝ ` ` 𝑇 𝑡 “1 𝑇 𝑀 𝑀 logp𝑇 {𝛿q , 𝑀 with probability at least 1 ´ 𝛿 Note that the dependence on the time horizon of the task is sub-linear. This is different from standard imitation learning regret bounds, which are often at least linear in the task horizon. The main reason is that our comparison benchmark 𝜋 ˚ does live in the space Π, whereas for DAgger, the expert policy may not reside in the same space. Noisy gradient estimate case. We now turn to the issue of estimating the gradient of ∇𝐽p𝜋q. We make the following simplifying assumption about the gradient estimation: 238 • The 𝜋 is parameterized by vector 𝜃 P R 𝑘 (such as a neural network). The parameterization is differentiable with respect to 𝜃 (Alternatively, we can view Π as a differentiable subspace of F , in which case we have H “ F ) • At each Update loop, the policy is rolled out 𝑚 times to collect the data, each trajectory has horizon length 𝐻 • For each visited state 𝑠 „ 𝑑 ℎ , the policy takes a uniformly random action 𝑎. The action space is finite with cardinality 𝐴. • The gradient ∇ℎ𝜃 is bounded by 𝐵 The gradient estimate is performed consistent with a generic policy gradient scheme, i.e., 𝐻 𝑚 𝐴ÿÿ 𝑗 𝑗 p𝑗 p ∇𝜋𝜃 p𝑎𝑖 |𝑠𝑖 , 𝜃q𝑄 ∇𝐽p𝜃q “ 𝑖 𝑚 𝑖“1 𝑗 “1 𝑗 p is the estimated cost-to-go (Sutton et al., 2000). where 𝑄 𝑖 Taking uniform random exploratory actions ensures that the samples are i.i.d. We can thus apply Bernstein’s inequality to obtain the bound between estimated gradient and the true gradient. Indeed, with probability at least 1 ´ 𝛿, we have that the following bound on the bias component-wise: › › p2𝜎 2 ` 2𝐴𝐻𝐵 3𝛽 q log 𝑘𝛿 ›p › ›∇𝐽p𝜃q ´ ∇𝐽p𝜃q› ď 𝛽 when 𝑚 ě 8 𝛽2 which leads to similar bound with respect to }¨}˚ (here we leverage the equivalence of norms in finite dimensional setting): ¸ ˜ › › p𝜎 2 ` 𝐴𝐻𝐵𝛽q log 𝑘𝛿 › › p 𝑡 › ď 𝛽 when 𝑚 “ 𝑂 ›∇𝑡 ´ ∇ ˚ 𝛽2 Applying union bound of this result over 𝑇 rounds of learning, and combining with the result from proposition (D.1.5), we have the following finite-sample guarantee in the simplifying policy gradient update. This is also the more detailed statement of theorem 5.4.2 in the main paper. Proposition D.1.6 (Finite-sample Guarantee under Noisy Gradient Updates and Projection Error). At each iteration, we perform policy gradient estimate using p𝜎 2 ` 𝐴𝐻𝐵𝛽q log 𝑇𝛿𝑘 𝑚 “ 𝑂p q trajectories and use DAgger algorithm to collect 𝑀 “ 𝛽2 239 c log 𝑇 {𝛿 𝑂p 𝜖 2 q`𝑂p 𝐻𝜖 q roll-outs. Setting the learning rate 𝜂 “ b ` 1 1 𝐻 `𝑀 ` 𝜎2 𝑇 logp𝑇 {𝛿q ˘ , 𝑀 after 𝑇 rounds of the algorithm, we have that ¨ d ˛ c 𝑇 ÿ logp𝑇{𝛿q ‚ 1 𝐻 1 ` ` 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď 𝑂 ˝𝜎 `𝛽 𝑇 𝑡 “1 𝑇 𝑀 𝑀 with probability at least 1 ´ 𝛿. Consequently, we also have the following regret bound: ˛ ¨ d c 𝑇 ÿ logp𝑇{𝛿q ‚ 𝐻 1 1 ` ` 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q ď𝑂 ˝𝜎 𝑇 𝑡 “1 𝑇 𝑀 𝑀 ˜ c ¸ logp𝑇 𝑘{𝛿q 𝐴𝐻 logp𝑇 𝑘{𝛿q `𝑂 𝜎 ` 𝑚 𝑚 holds with probability at least 1 ´ 𝛿, where again 𝐻 is the task horizon, 𝐴 is the cardinality of action space, and 𝑘 is the dimension of function class Π’s parameterization. Proof. (For both proposition (D.1.6) and (D.1.5)). The results follow by taking the inequality from equation (D.16), and by solving for 𝜖 and 𝛽 explicitly in terms of relevant quantities. Based on the specification of 𝑀 and 𝑚, we obtain the necessary precision for each round of learning in terms of number of trajectories: logp𝑘{𝛿q 𝐴𝐻𝐵 logp𝑘{𝛿q ` q 𝑚 𝑚 c logp1{𝛿q 𝐻 𝜖 “ 𝑂p ` q 𝑀 𝑀 b ` ˘ Setting the learning rate 𝜂 “ 𝜎12 𝑇1 ` 𝜖 and rearranging the inequalities lead to the desired bounds. 𝛽 “ 𝑂p𝜎 The regret bound depends on the variance 𝜎 2 of the policy gradient estimates. It is well-known that vanilla policy gradient updates suffer from high variance. We instead use functional regularization technique, based on CORE-RL, in the practical implementation of our algorithm. The CORE-RL subroutine has been demonstrated to reduce the variance in policy gradient updates (Cheng et al., 2019e). 240 Defining a consistent approximation of ∇H 𝐽p𝜋q - Proof of Proposition 5.4.3 We are using the notion of Fréchet derivative to define gradient of differentiable functional. Note that while Gateaux derivative can also be utilized, Fréchet derivative ensures continuity of the gradient operator that would be useful for our analysis. Definition D.1.6 (Fréchet gradient). A bounded linear operator ∇ : H ÞÑ H is 𝐽 p ℎ`𝑔 q´𝐽 p ℎq´x∇𝐽 p ℎq,𝑔 y called Fréchet functional gradient of 𝐽 at ℎ P H if lim “0 }𝑔 } }𝑔}Ñ0 We make the following assumption about H and F . One interpretation of this assumption is that the space of policies Π and F that we consider have the property that a programmatic policy 𝜋 P Π can be well-approximated by a large space of neural policies 𝑓 P F . Assumption 6. 𝐽 is Fréchet differentiable on H . 𝐽 is also differentiable on the s “ H) restricted subspace F . And F is dense in H (i.e., the closure F It is then clear that @ 𝑓 P F the Fréchet gradient ∇F 𝐽p 𝑓 q, restricted to the subspace F is equal to the gradient of 𝑓 in the ambient space H (since Fréchet gradient is unique). In general, given 𝜋 P Π and 𝑓 P F , 𝜋 ` 𝑓 is not necessarily in F . However, the restricted gradient on subspace F of 𝐽p𝜋 ` 𝑓 q can be defined asymptotically. Proposition D.1.7. Fixing a policy 𝜋 P Π, define a sequence of policies 𝑓 𝑘 P F , 𝑘 “ 1, 2, . . . that converges to 𝜋: lim 𝑘 Ñ8 } 𝑓 𝑘 ´ 𝑔} “ 0, we then have lim 𝑘 Ñ8 }∇F 𝐽p 𝑓 𝑘 q ´ ∇H 𝐽p𝜋q}˚ “ 0 Proof. Since Fréchet derivative is a continuous linear operator, we have lim 𝑘 Ñ8 }∇H 𝐽p 𝑓 𝑘 q ´ ∇H 𝐽p𝜋q}˚ “ 0. By the reasoning above, for 𝑓 P F , the gradient ∇F 𝐽p 𝑓 q defined via restriction to the space F does not change compared to ∇H 𝐽p 𝑓 q, the gradient defined over the ambient space H . Thus we also have lim 𝑘 Ñ8 }∇F 𝐽p 𝑓 𝑘 q ´ ∇H 𝐽p𝜋q}˚ “ 0. By the same argument, we also have that for any given 𝜋 P Π and 𝑓 P F , even if 𝜋 ` 𝑓 R F , the gradient ∇F 𝐽p𝜋 ` 𝑓 q with respect to the F can be approximated similarly. Note that we are not assuming 𝐽p𝜋q to be differentiable when restricting to the policy subspace Π. 241 Theoretical motivation for Algorithm 6 - Proof of Proposition 5.4.4 and 5.4.5 We consider the case where Π is not differentiable by parameterization. Note that this does not preclude 𝐽p𝜋q for 𝜋 P Π to be differentiable in the non-parametric function space. Two complications arise compared to our previous approximate mirror descent procedure. First, for each 𝜋 P Π, estimating the gradient ∇𝐽p𝜋q (which may not exist under certain parameterization, per section 5.4) can become much more difficult. Second, the update rule ∇𝑅p𝜋q ´ ∇F 𝐽p𝜋q may not be in the dual space of F , as in the simple case where Π Ă F , thus making direct gradient update in the F space inappropriate. Assumption 7. 𝐽 is convex in H . By convexity of 𝐽 in H , sub-gradients B𝐽pℎq exists for all ℎ P H . In particular, B𝐽p𝜋q exists for all 𝜋 P Π. Note that B𝐽p𝜋q reflects sub-gradient of 𝜋 with respect to the ambient policy space H . We will make use of the following equivalent perspective to mirror descent(Beck and Teboulle, 2003), which consists of two-step process for each iteration 𝑡 1. Solve for ℎ𝑡 `1 “ arg minℎPH 𝜂xB𝐽p𝜋𝑡 q, ℎy ` 𝐷 𝑅 pℎ, 𝜋𝑡 q 2. Solve for 𝜋𝑡 `1 “ arg min𝜋PΠ 𝐷 𝑅 p𝜋, ℎ𝑡 `1 q We will show how this version of the algorithm motivates our main algorithm. Consider step 1 of the main loop of Propel, where given a fixed 𝜋 P Π, the optimization problem within H is pOBJECTIVE_1q “ min 𝜂xB𝐽p𝜋q, ℎy ` 𝐷 𝑅 pℎ, 𝜋q ℎ PH (D.17) Due to convexity of H and the objective, problem pOBJECTIVE_1q is equivalent to: pOBJECTIVE_1q “ minxB𝐽p𝜋q, ℎy s.t. 𝐷 𝑅 pℎ, 𝜋q ď 𝜏 (D.18) (D.19) where 𝜏 depends on 𝜂. Since 𝜋 is fixed, this optimization problem can be relaxed by choosing 𝜆 P r0, 1s, and a set of candidate policies ℎ “ 𝜋 ` 𝜆 𝑓 , for all 𝑓 P F , such that 𝐷 𝑅 pℎ, 𝜋q ď 𝜏 is satisfied (Selection of 𝜆 is possible with bounded spaces). Since this constraint set is potentially a restricted set compared to the space of 242 policies satisfying inequality (D.19), the optimization problem (D.17) is relaxed into: pOBJECTIVE_1q ď pOBJECTIVE_2q “ minxB𝐽p𝜋q, 𝜋 ` 𝜆 𝑓 y 𝑓 PF (D.20) Due to convexity property of 𝐽, we have xB𝐽p𝜋q, 𝜆 𝑓 y “ xB𝐽p𝜋q, 𝜋 ` 𝜆 𝑓 ´ 𝜋qy ď 𝐽p𝜋 ` 𝜆 𝑓 q ´ 𝐽p𝜋q (D.21) The original problem OBJECTIVE_1 is thus upper bounded by: ` ˘ min 𝜂xB𝐽p𝜋q, ℎqy ` 𝐷 𝑅 pℎ, 𝜋q ď min 𝐽 𝜋 ` 𝜆 𝑓 ´ 𝐽p𝜋q ` xB𝐽p𝜋q, 𝜋y ℎ PH 𝑓 PF Thus, a relaxed version of original optimization problem OBJECTIVE_1 can be obtained by miniziming 𝐽p𝜋 ` 𝜆 𝑓 q over 𝑓 P F (note that 𝜋 is fixed). This naturally motivates using functional regularization technique, such as CORE-RL algorithm (Cheng et al., 2019e), to update the parameters of differentiable function 𝑓 via policy gradient descent update: 𝑓 1 “ 𝑓 ´ 𝜂𝜆∇F 𝜆𝐽p𝜋 ` 𝜆 𝑓 q where the gradient of 𝐽 is taken with respect to the parameters of 𝑓 (neural networks). This is exactly the update step in algorithm 6 (also similar to iterative updte of CORERL algorithm), where the neural network policy is regularized by a prior controller 𝜋. Statement and Proof of Proposition 5.4.5 Proposition D.1.8 (Regret bound for the relaxed optimization objective). Assuming 𝐽pℎq is 𝐿-strongly smooth over H , i.e., ∇H 𝐽pℎq is 𝐿-Lipschitz continuous, approximating Update to the expected regret bound: b Alg. 6 leads ¯ ” ř ı H by Update ´ 𝐹 per 𝑇 E 𝑇1 𝑡 “1 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝜆𝜎 𝑇1 ` 𝜖 ` 𝜆2 𝐿 2 Proof. Instead of focusing on the bias of the gradient estimate ∇H 𝐽p𝜋q, we will shift our focus on the alternative proximal formulation of mirror descent, under optimization and projection errors. In particular, at each iteration 𝑡, let ℎ𝑡˚`1 “ arg minℎPH 𝜂x∇𝐽p𝜋𝑡 q, ℎy ` 𝐷 𝑅 pℎ, 𝜋𝑡 q and let the optimization error be defined as 𝛽𝑡 where ∇𝑅pℎ𝑡 `1 q “ ∇𝑅pℎ𝑡˚`1 q ` 𝛽𝑡 . Note here that this is different from (but related to) the notion of bias from gradient estimate of ∇𝐽p𝜋q used in theorem 5.4.1 and theorem D.1.2. The projection error from imitation learning procedure is defined 243 similarly to theorem 5.4.1: 𝜋𝑡˚`1 “ arg min𝜋PΠ 𝐷 𝑅 p𝜋, ℎ𝑡 `1 q is the true projection, › › › › and ›𝜋𝑡 `1 ´ 𝜋𝑡˚`1 › ď 𝜖. We start with similar bounding steps to the proof of theorem 5.4.1: 1 x∇𝑅pℎ𝑡˚`1 q ´ ∇𝑅p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y 𝜂 1 “ px∇𝑅pℎ𝑡 `1 q ´ ∇𝑅p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y ´ x𝛽𝑡 , 𝜋𝑡 ´ 𝜋yq 𝜂 1 1 “ p𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, ℎ𝑡 `1 q ` 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 qq ` x𝛽𝑡 , 𝜋𝑡 ´ 𝜋y 𝜂 𝜂 looooooooooooooooooooooooooomooooooooooooooooooooooooooon loooooomoooooon x∇𝐽p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y “ component_1 component_2 (D.22) As seen from the proof of theorem D.1.2, component_1 can be upperbounded by: 1` 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ` 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡˚`1 q 𝜂 loooooooooooooomoooooooooooooon loooooooooooooooomoooooooooooooooon telescoping ˚ ´𝐷 𝑅 p𝜋𝑡 `1 , ℎ𝑡 `1 q ` 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q looooooooooooooooooomooooooooooooooooooon projection error ˘ relative improvement The bound on projection error is identical to theorem D.1.2: 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡˚`1 q ď 𝜖 𝐿 𝑅 𝐷 (D.23) The bound on relative improvement is slightly different: 𝐷 𝑅 p𝜋𝑡 , ℎ𝑡 `1 q ´ 𝐷 𝑅 p𝜋𝑡˚`1 , ℎ𝑡 `1 q “ 𝑅p𝜋𝑡 q ´ 𝑅p𝜋𝑡˚`1 q ` x∇𝑅pℎ𝑡 `1 q, 𝜋𝑡˚`1 ´ 𝜋𝑡 y “ 𝑅p𝜋𝑡 q ´ 𝑅p𝜋𝑡˚`1 ` x∇𝑅pℎ𝑡˚`1 q, 𝜋𝑡˚`1 ´ 𝜋𝑡 yq ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋𝑡 y ›2 𝛼› ď x∇𝑅p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋𝑡˚`1 y ´ ›𝜋𝑡˚`1 ´ 𝜋𝑡 › 2 ˚ ˚ ` x∇𝑅pℎ𝑡 `1 q, 𝜋𝑡 `1 ´ 𝜋𝑡 y ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋𝑡 y ›2 𝛼› (D.24) “ ´𝜂x∇𝐽H p𝜋𝑡 q, 𝜋𝑡˚`1 ´ 𝜋𝑡 y ´ ›𝜋𝑡˚`1 ´ 𝜋𝑡 › ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋𝑡 y 2 𝜂2 }∇H 𝐽p𝜋𝑡 q}2˚ ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋𝑡 y ď 2𝛼 𝜂2 2 𝐿 ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋𝑡 y ď (D.25) 2𝛼 𝐽 Note here that the gradient ∇H 𝐽p𝜋𝑡 q is not the result of estimation. Combining equations (D.22), (D.23), (D.24), (D.25), we have: ˘ 𝜂2 2 1` x∇𝐽p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y ď 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ` 𝜖 𝐿 𝑅 𝐷 ` 𝐿 𝐽 ` x𝛽𝑡 , 𝜋𝑡˚`1 ´ 𝜋y 𝜂 2𝛼 (D.26) 244 Next, we want to bound 𝛽𝑡 . Choose regularizer 𝑅 to be 12 }¨}2 (consistent with the pseudocode in algorithm 6). We have that: ℎ𝑡˚`1 “ arg min 𝜂x∇𝐽p𝜋𝑡 q, ℎy ` ℎPH 1 }ℎ ´ 𝜋𝑡 }2 2 which is equivalent to: ℎ𝑡˚`1 “ 𝜋𝑡 ` arg min 𝜂x∇𝐽p𝜋𝑡 q, 𝑓 y ` 𝑓 PF 1 } 𝑓 }2 2 Let 𝑓𝑡˚`1 “ arg min 𝑓 PF 𝜂x∇𝐽p𝜋𝑡 q, 𝑓 y ` 12 } 𝑓 }2 . Taking the gradient over 𝑓 , we can see that 𝑓𝑡˚`1 “ ´𝜂∇𝐽p𝜋𝑡 q. Let 𝑓𝑡 `1 be the minimizer of min 𝑓 PF 𝐽p𝜋𝑡 `𝜆 𝑓 q. We then have ℎ𝑡˚`1 “ 𝜋𝑡 ` 𝑓𝑡˚`1 and ℎ𝑡 `1 “ 𝜋 ` 𝜆 𝑓𝑡 `1 . Thus 𝛽𝑡 “ ℎ𝑡 `1 ´ ℎ𝑡˚`1 “ 𝑓𝑡 `1 ´ 𝑓𝑡˚`1 . On one hand, we have 𝐽p𝜋𝑡 ` 𝜆 𝑓𝑡 `1 q ď 𝐽p𝜋𝑡 ` 𝜔 𝑓𝑡˚`1 q ď 𝐽p𝜋𝑡 q ` x∇𝐽p𝜋𝑡 q, 𝜔 𝑓𝑡˚`1 y ` 𝐿 ›› ˚ ››2 𝜔 𝑓 𝑡 `1 2 due to optimality of 𝑓𝑡 `1 and strong smoothness property of 𝐽. On the other hand, since 𝐽 is convex, we also have the first-order condition: 𝐽p𝜋𝑡 ` 𝜆 𝑓𝑡 `1 q ě 𝐽p𝜋𝑡 q ` x∇𝐽p𝜋𝑡 q, 𝜆 𝑓𝑡 `1 y Combine with the inequality above, and subtract 𝐽p𝜋𝑡 q from both sides, and using the relationship 𝑓𝑡˚`1 “ ´𝜂∇𝐽p𝜋𝑡 q, we have that: 1 𝐿𝜔2 ›› ˚ ››2 1 𝑓 𝑡 `1 x´ 𝑓𝑡˚`1 , 𝜆 𝑓𝑡 `1 y ď x´ 𝑓𝑡˚`1 , 𝜔 𝑓𝑡˚`1 y ` 𝜂 𝜂 2 2 𝜆 Since this is true @𝜔, rearrange and choose 𝜔 such that 𝜔𝜂 ´ 𝐿𝜔 2 “ ´ 2𝜂 , namely 𝜔“ ? 1´ 1´𝜆𝜂𝐿 , and complete the square, we can establish the bound that: 𝐿𝜂 › › › 𝑓𝑡 `1 ´ 𝑓 ˚ › ď 𝜂p𝜆𝐿q2 𝐵 𝑡 `1 (D.27) for 𝐵 the upperbound on } 𝑓𝑡 `1 }. We thus have }𝛽𝑡 } “ 𝑂p𝜂p𝜆𝐿q2 q. Plugging the result from equation D.27 into RHS of equation D.26, we have: ˘ 𝜂2 2 ˘ ` 1` 𝐷 𝑅 p𝜋, 𝜋𝑡 q ´ 𝐷 𝑅 p𝜋, 𝜋𝑡 `1 q ` 𝜖 𝐿 𝑅 𝐷 ` 𝐿 𝐽 ` 𝜂p𝜆𝐿q2 𝐵 𝜂 2𝛼 (D.28) Since 𝐽 is convex in H , we b have 𝐽p𝜋𝑡 q ´ 𝐽p𝜋q ď x∇𝐽p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y. Similar to theorem 5.4.1, setting 𝜂 “ 𝜆21𝜎2 p 𝑇1 ` 𝜖q and taking expectation on both sides, we have: « ff c 𝑇 ` ˘ 1ÿ 1 E 𝐽p𝜋𝑡 q ´ 𝐽p𝜋 ˚ q “ 𝑂 𝜆𝜎 ` 𝜖 ` 𝜆2 𝐿 2 (D.29) 𝑇 𝑡 “1 𝑇 x∇𝐽p𝜋𝑡 q, 𝜋𝑡 ´ 𝜋y ď 245 Note that unlike regret bound from theorem 5.4.1 under general bias, variance of gradient estimate and projection error, 𝜎 2 here explicitly refers to the bound on neural-network based policy gradient variance. The variance reduction of 𝜆𝜎, at the expense of some bias, was also similarly noted in a recent functional regularization technique for policy gradient (Cheng et al., 2019e). D.2 Additional Experimental Results and Details TORCS We generate controllers for cars in The Open Racing Car Simulator (Torcs) (Wymann et al., 2014). In its full generality Torcs provides a rich environment with input from up to 89 sensors, and optionally the 3D graphic from a chosen camera angle in the race. The controllers have to decide the values of 5 parameters during game play, which correspond to the acceleration, brake, clutch, gear and steering of the car. Apart from the immediate challenge of driving the car on the track, controllers also have to make race-level strategy decisions, like making pit-stops for fuel. A lower level of complexity is provided in the Practice Mode setting of TORCS. In this mode all race-level strategies are removed. Currently, so far as we know, state-ofthe-art Drl models are capable of racing only in Practice Mode, and this is also the environment that we use. Here we consider the input from 29 sensors, and decide values for the acceleration, steering, and braking actions. We chose a suite of tracks that provide varying levels of difficulty for the learning algorithms. In particular, for the tracks Ruudskogen and Alpine-2, the Ddpg agent is unable to reliably learn a policy that would complete a lap. We perform the experiments with twenty-five random seeds and report the median lap time over these twenty-five trials. However we note that the Torcs simulator is not deterministic even for a fixed random seed. Since we model the environment as a Markov Decision Process, this non-determinism is consistent with our problem statement. For our Deep Reinforcement Learning agents we used standard open source implementations (with pre-tuned hyper-parameters) for the relevant domain. All experiments were conducted on standard workstation with a 2.5 GHz Intel Core i7 CPU and a GTX 1080 Ti GPU card. The code for the Torcs experiments can be found at: https://bitbucket.org/averma8053/propel 246 In Table D.1 we show the lap time performance and crash ratios of Propel agents initialized with neural policies obtained via Ddpg. As discussed in Section 5.5, Ddpg often exhibits high variance across trials and this adversely affects the performance of the Propel agents when they are initialized via Ddpg. In Table D.2 we show generalization results for the IppgTree agent. As noted in Section 5.5, the generalization results for IppgTree are in between those of Ddpg and IppgProgram. Verified Smoothness Property. For the program given in Figure 2 we proved using symbolic verification techniques, that @𝑘, ř 𝑘 `5 𝑖 “ 𝑘 }peekp𝑠rRPMs, 𝑖 ` 1q ´ peekp𝑠rRPMs, 𝑖q} ă 0.003 ùñ }peekp𝑎rAccels, 𝑘 ` 1q ´ peekp𝑎rAccels, 𝑘q} ă 0.63. Here the function peekp., 𝑖q takes in a history/sequence of sensor or action values and returns the value at position 𝑖, . Intuitively, the above logical implication means that if the sum of the consecutive differences of the last six RPM sensor values is less than 0.003, then the acceleration actions calculated at the last and penultimate step will not differ by more than 0.63. Table D.1: Performance results in Torcs of Propel agents initialized with neural policies obtained via Ddpg, over 25 random seeds. Each entry is formatted as Laptime / Crash-ratio, reporting median lap time in seconds over all the seeds (lower is better) and ratio of seeds that result in crashes (lower is better). A lap time of Cr indicates the agent crashed and could not complete a lap for more than half the seeds. Length G-Track 3186m E-Road 3260m Aalborg 2588m Ruudskogen 3274m Alpine-2 3774m IppgProgram-Ddpg IppgTree-Ddpg 97.76/.12 78.47/0.16 108.06/.08 85.46/.04 140.48/.48 Cr / 0.56 Cr / 0.68 Cr / 0.68 Cr / 0.92 Cr / 0.92 Classic Control We present results from two classic control problems, Mountain-Car (with continuous actions) and Pendulum, in Table D.3. We use the OpenAI Gym implementations of these environments. More information about these environments can be found at the links: MountainCar and Pendulum. In Mountain-Car the goal is to drive an under-powered car up the side of a mountain in as few time-steps as possible. In Pendulum, the goal is to swing a pendulum 247 Table D.2: Generalization results in Torcs for IppgTree, where rows are training and columns are testing tracks. Each entry is formatted as IppgProgram / DDPG, and the number reported is the median lap time in seconds over all the seeds (lower is better). Cr indicates the agent crashed and could not complete a lap for more than half the seeds. G-Track E-Road Aalborg Ruudskogen Alpine-2 G-Track E-Road Aalborg Ruudskogen Alpine-2 84 111 154 Cr 95 Cr Cr 276 Cr Cr Cr Cr Cr Cr Cr Cr Cr Cr Cr Cr - Table D.3: Performance results in Classic Control problems. Higher scores are better. Prior Ddpg Trpo Ndps Viper IppgProgram IppgTree MountainCar Pendulum 00.59 ˘ 0.00 97.16 ˘ 3.21 93.03 ˘ 1.86 66.98 ˘ 3.11 64.86 ˘ 3.28 95.63 ˘ 1.02 96.56 ˘ 2.81 ´875.53 ˘ 0.00 ´132.70 ˘ 6.44 ´131.54 ˘ 4.56 ´435.71 ˘ 4.83 ´394.11 ˘ 4.97 ´187.71 ˘ 2.35 ´139.09 ˘ 3.31 up so that it stays upright. In both the environments an episode terminates after a maximum of 200 time-steps. In Table D.3 we report the mean and standard deviation, over twenty-five random seeds, of the average scores over 100 episodes for the listed agents and environments. In Figure D.1 and Figure D.2 we show the improvements made over the prior by the IppgProgram agent in MountainCar and Pendulum respectively, with each iteration of the Propel algorithm. 248 100 Pendulum Score Improvement Score Improvement MountainCar 80 60 40 20 600 400 200 0 0 0 1 2 3 Iterations Figure D.1: Score improvements in the MountainCar environment over iterations of IppgProgram. 0 1 2 3 Iterations Figure D.2: Score improvements in the Pendulum environment over iterations of IppgProgram. 249 Appendix E APPENDIX TO CHAPTER 6 E.1 Proofs for Chapter 6 I Proof of Theorem 6.4.2. The first term 𝑇𝐶full should be obvious as the expert inspects the agent’s overall behavior in each episode. Whenever something goes L wrong in an episode, the expert labels the whole trajectory, incurring 𝐶full each time. The remaining work is to bound the number of episodes where agent makes one or more mistakes. This quantity is bounded by the number of total mistakes made by the halving algorithm, which is at most the logarithm of the number of candidate ` ˘ functions (policies), log |Πfull | “ log |M||Πlo ||G | “ log |M| ` |G| log |Πlo |. This completes the proof. I Proof of Theorem 6.4.1. Similar to the proof of Theorem 6.4.2, the first term 𝑇𝐶full is obvious. The second term corresponds to the situation where Inspectfull finds issues. According to Algorithm 9, the expert then labels the subgoals and also inspects whether each subgoal is accomplished successfully, which incurs L ` 𝐻 𝐶 I cost each time. The number of times that this situation happens 𝐶hi hi lo is bounded by (a) the number of times that a wrong subgoal is chosen, plus (b) the number of times that all subgoals are good but at least one of the subpolicies fails to accomplish the subgoal. Situation (a) occurs at most log |M| times. In situation (b), the subgoals chosen in the episode must come from Gopt , and for each of these subgoals the halving algorithm makes at most log |Πlo | mistakes. The last term corresponds to cost of Labello operations. This only occurs when the metacontroller chooses a correct subgoal but the corresponding subpolicy fails. Similar to previous analysis, this situation occurs at most log |Πlo | for each “good” subgoal (𝑔 P Gopt ). This completes the proof. E.2 Additional Experimental Details In our experiments, success rate and external rewards are reported as the trailing average over previous 100 episodes of training. For hierarchical imitation learning experiments in maze navigation domain, the success rate is only measured on separate test environments not used for training. In addition to experimental results, in this section we describe our mechanism for 250 hg-DAgger/Q vs. h-DQN (Maze Navigation) 100% 400K success rate 80% 300K 250K 60% hg-DAgger/Q success rate hg-DAgger/Q HI-level IL cost h-DQN success rate h-DQN HI-level RL samples 40% 20% 0% 0K 50K 200K 150K 100K HI-level cost (RL or IL) 350K 50K 0K 100K 150K 200K 250K 300K 350K 400K RL samples at LO-level Figure E.1: Maze navigation: hybrid IL-RL (full task) versus h-DQN (with 50% head-start). subgoal detection / terminal predicate for Montezuma’s Revenge and how the Maze Navigation environments are created. Network architectures from our experiments are in Tables E.1 and E.2. Maze Navigation Domain We compare hg-DAgger/Q with the hierarchical reinforcement learning baseline (h-DQN, (Kulkarni et al., 2016)) with the same network architecture for the metacontroller and subpolicies as hg-DAgger/Q and similarly enhanced 𝑄-learning procedure. Similar to the Montezuma’s Revenge domain, h-DQN does not work well for the maze domain. At the hi level, the planning horizon of 10–12 with 4–5 possible subgoals in each step is prohibitively difficult for the hi-level reinforcement learner and we were not able to achieve non-zero rewards within in any of our experiments. To make the comparison, we attempted to provide additional advantage to the h-DQN algorithm by giving it some head-start, so we ran h-DQN with 50% reduction in the horizon, by giving the hierarchical learner the optimal execution of the first half of the trajectory. The resulting success rate is in Figure E.1. Note that the hybrid IL-RL does not get the 50% advantage, but it still quickly outperforms h-DQN, 251 Figure E.2: Maze navigation. Sample random instances of the maze domain (different from main text). The 17 ˆ 17 pixel representation of the maze is used as input for neural network policies. which flattens out at 30% success rate. Creating Maze Navigation Environments We create 2000 maze navigation environments, 1000 of which are used for training and 1000 maps are used for testing. The comparison results for maze navigation (e.g., Figure 6.2) are all based on randomly selected environments among 1000 test maps. See Figure E.2 for additional examples of the environments created. For each map (environment instance), we start with a 17ˆ17 grid, which are divided into 4ˆ4 room structure. Initially, no door exists in between rooms. To create an instance of the maze navigation environment, the goal block (yellow) and the starting position are randomly selected (accepted as long as they are not the same). Next, we randomly select a wall separating two different room and replace a random red block (lava) along this wall with a door (black cell). This process continues until two conditions are satisfied: • There is a feasible path between the starting location and the goal block (yellow) • The minimum distance between start to goal is at least 40 steps. The optimal path can be constructed using graph search Each of the 2000 environments create must satisfy both conditions. The expert labels for each environment come from optimal policy computed via value iteration (which is fast based on tabular representation of the given grid world). Hyperparameters for Maze Navigation The network architecture used for maze navigation is described in Table E.1. The only difference between subgoal policy networks and metacontroller network is the 252 Figure E.3: Montezuma’s Revenge: Screenshots of the environment with 4 designated subgoals in sequence. number of output class (4 actions versus 5 subgoals). For our hierarchical imitation learning algorithms, we also maintain a small network along each subgoal policy for subgoal termination classification (one can also view the subgoal termination classifier as an extra head of the subgoal policy network). The contextual input (state) to the policy networks consists of 3-channel pixel representation of the maze environment. We assign different (fixed) values to goal block, agent location, agent’s trail and lava blocks. In our hierarchical imitation learning implementations, the base policy learner (DAgger and behavior cloning) update the policies every 100 steps using stochastic optimization. We use Adam optimizer and learning rate of 0.0005. Table E.1: Network Architecture—Maze Domain 1: Convolutional Layer 2: Convolutional Layer 3: Max Pooling Layer 32 filters, kernel size 3, stride 1 32 filters, kernel size 3, stride 1 pool size 2 4: Convolutional Layer 5: Convolutional Layer 6: Max Pooling Layer 64 filters, kernel size 3, stride 1 64 filters, kernel size 3, stride 1 pool size 2 7: Fully Connected Layer 256 nodes, relu activation 8: Output Layer softmax activation (dimension 4 for subpolicy, dimension 5 for meta-controller) Montezuma’s Revenge Although the imitation learning component tends to be stable and consistent, the samples required by the reinforcement learners can vary between experiments with identical hyperparameters. In this section, we report additional results of our hybrid algorithm for the Montezuma’s Revenge domain. For the implementation of our hybrid algorithm on the game Montezuma’s Revenge, 253 200 100 0 0.0M 0.5M 1.0M 1.5M 2.0M LO-level reinforcement learning samples Subgoals Completion over 100 Trials (hg-DAgger/Q vs. h-DQN) hg-DAgger/Q h-DQN 4.0 3.5 3.0 100 2.5 2.0 1.5 1.0 Learning Curve for Getting Key (hg-DAgger/Q vs. h-DQN over 100 trials) hg-DAgger/Q 3rd quartile hg-DAgger/Q median h-DQN median 80 external rewards external rewards 300 subgoals completed (out of 4) Best 10 Trials (hg-DAgger/Q vs. h-DQN) hg-DAgger/Q h-DQN 400 60 40 20 0.5 0.0 0.0M 0.5M 1.0M 1.5M 2.0M 2.5M 3.0M 3.5M 4.0M 0 0.0M LO-level reinforcement learning samples 0.2M 0.4M 0.6M 0.8M 1.0M LO-level reinforcement learning samples Figure E.4: Montezuma’s revenge: hybrid IL-RL versus hierarchical RL. (Left) Median reward, min and max across the best 10 trials. The agent completes the first room in less than 2 million samples. The shaded region corresponds to min and max of the best 10 trials. (Middle) Median, first and third quartile of subgoal completion rate across 100 trials. The shaded region corresponds to first and third quartile. (Right) Median, first and third quartile of reward across 100 trials. The shaded region corresponds to first and third quartile. h-DQN only considers the first two subgoals to simplify the learning task. 1K 2K 3K 4K 5K 6K 7K episode (HI-level labeling cost) 8K Learning Progression (random trial) 100% 80% 60% 40% 20% 0% 1000K 800K 600K 400K 200K 0K 0K Learning Progression (random trial) success rate 100% 80% 60% 40% 20% 0% 500K 400K 300K 200K 100K 0K 0K Subgoal 1 Subgoal 2 (key) Subgoal 3 Subgoal 4 (door) 1K 2K 3K 4K episode (HI-level labeling cost) 5K LO-level samples Subgoal 1 Subgoal 2 (key) Subgoal 3 Subgoal 4 (door) LO-level samples LO-level samples success rate Learning Progression (random trial) success rate 100% 80% 60% 40% 20% 0% 1800K 1600K 1400K 1200K 1000K 800K 600K 400K 200K 0K 0K Subgoal 1 Subgoal 2 (key) Subgoal 3 Subgoal 4 (door) 1K 2K 3K 4K 5K 6K 7K 8K episode (HI-level labeling cost) Figure E.5: Montezuma’s revenge: Learning progression of Algorithm 3 in solving the entire first room. The figures show three randomly selected successful trials. we decided to limit the computation to 4 million frames for the lo-level reinforcement learners (in aggregate across all 4 subpolicies). Out of 100 experiments, 81 out of 100 successfully learn the first 3 subpolicies, 89 out of 100 successfully learn the first 2 subpolicies. The last subgoal (going from the bottom of the stairs to open the door) proved to be the most difficult and almost half of our experiments did not manage to finish learning the fourth subpolicy within the 4 million frame limit (see Figure E.4 middle pane). The reason mainly has to do with the longer horizon of subgoal 4 compared to other three subgoals. Of course, this is a function of the design of subgoals and one can always try to shorten the horizon by introducing intermediate subgoals. However, it is worth pointing out that even as we limit the h-DQN baseline to only 2 subgoals (up to getting the key), the h-DQN baseline generally tends to underperform our proposed hybrid algorithm by a large margin. Even with the given advantage we confer to our implementation of h-DQN, all of the h-DQN experiments failed to successfully master the second subgoal (getting the key). It is instructive to also 254 25 HI-level Expert Cost over 100 Trials (hg-DAgger/Q) frequency 20 15 10 5 0 4K 6K 8K 10K 12K 14K 16K 18K HI-level expert cost Figure E.6: Montezuma’s Revenge: Number of HI-level expert labels. Distribution of Hi-level expert labels needed across 100 trials; the histogram excludes 6 outliers whose number of labels exceeds 20K for ease of visualization examine the sample complexity associated with getting the key (the first positive external reward, see Figure E.4 right pane). Here the horizon is sufficiently short to appreciate the difference between having expert feedback at the hi level versus relying only on reinforcement learning to train the meta-controller. The stark difference in learning performance (see Figure E.4 right) comes from the fact that the hi-level expert advice effectively prevents the lo-level reinforcement learners from accumulating bad experience, which is frequently the case for h-DQN. The potential corruption of experience replay buffer also implies at in our considered setting, learning with hierarchical DQN is no easier compared to flat DQN learning. Hierarchical DQN is thus susceptible to collapsing into the flat learning version. Subgoal Detectors for Montezuma’s Revenge In principle, the system designer would select the hierarchical decomposition that is most convenient for giving feedback. For Montezuma’s Revenge, we set up four subgoals and automatic detectors that make expert feedback trivial. The subgoals are landmarks that are described by small rectangles. For example, the door subgoal (subgoal 4) would be represented by a patch of pixel around the right door (see 255 Figure E.3 right). We can detect the correct termination / attainment of this subgoal by simply counting the number of pixels inside of the pre-specified box that has changed in value. Specifically in our case, subgoal completion is detected if at least 30% of pixels in the landmark’s detector box changes. Hyperparameters for Montezuma’s Revenge Table E.2: Network Architecture—Montezuma’s Revenge 1: Conv. Layer 2: Conv. Layer 3: Conv. Layer 32 filters, kernel size 8, stride 4, relu 64 filters, kernel size 4, stride 2, relu 64 filters, kernel size 3, stride 1, relu 4: Fully Connected Layer 512 nodes, relu, normal initialization with std 0.01 5: Output Layer linear (dimension 8 for subpolicy, dimension 4 for meta-controller) Neural network architecture used is similar to (Kulkarni et al., 2016). One difference is that we train a separate neural network for each subgoal policy, instead of maintaining a subgoal encoding as part of the input into a policy neural network that shares representation for multiple subgoals jointly. Empirically, sharing representation across multiple subgoals causes the policy performance to degrade we move from one learned subgoal to the next (a phenomenon of catastrophic forgetting in deep learning literature). Maintaining each separate neural network for each subgoal ensures the performance to be stable across subgoal sequence. The metacontroller policy network also has similar architecture. The only difference is the number of output (4 output classes for metacontroller, versus 8 classes (actions) for each lo-level policy). For training the lo-level policy with 𝑄-learning, we use DDQN (Van Hasselt et al., 2016) with prioritized experience replay (Schaul et al., 2015b) (with prioritization exponent 𝛼 “ 0.6, importance sampling exponent 𝛽0 “ 0.4). Similar to previous deep reinforcement learning work applied on Atari games, the contextual input (state) consists of four consecutive frames, each converted to grey scale and reduced to size 84ˆ 84 pixels. Frame skip parameter as part of the Arcade Learning Environment is set to the default value of 4. The repeated action probability is set to 0, thus the Atari environment is largely deterministic. The experience memory has capacity of 500K. The target network used in 𝑄-learning is updated every 2000 steps. For stochastic 256 optimization, we use rmsProp with learning rate of 0.0001, with mini-batch size of 128. E.3 Additional Related Work Imitation Learning. Another dichotomy in imitation learning, as well as in reinforcement learning, is that of value-function learning versus policy learning. The former setting (Abbeel and Ng, 2004; Ziebart et al., 2008) assumes that the optimal (demonstrated) behavior is induced by maximizing an unknown value function. The goal then is to learn that value function, which imposes a certain structure onto the policy class. The latter setting (Daumé III et al., 2009; Ross et al., 2011b; Ho and Ermon, 2016) makes no such structural assumptions and aims to directly fit a policy whose decisions well imitate the demonstrations. This latter setting is typically more general but often suffers from higher sample complexity. Our approach is agnostic to this dichotomy and can accommodate both styles of learning. Some instantiations of our framework allow for deriving theoretical guarantees, which rely on the policy learning setting. Sample complexity comparison between imitation learning and reinforcement learning has not been studied much in the literature, perhaps with the exception of the recent analysis of AggreVaTeD (Sun et al., 2017). Hierarchical Reinforcement Learning. Feudal RL is another hierarchical framework that is similar to how we decompose the task hierarchically (Dayan and Hinton, 1993; Dietterich, 2000; Vezhnevets et al., 2017). In particular, a feudal system has a manager (similar to our hi-level learner) and multiple submanagers (similar to our lo-level learners), and submanagers are given pseudo-rewards which define the subgoals. Prior work in feudal RL use reinforcement learning for both levels; this can require a large amount of data when one of the levels has a long planning horizon, which we demonstrate in our experiments. In contrast, we propose a more general framework where imitation learners can be used to substitute reinforcement learners to substantially speed up learning, whenever the right level of expert feedback is available. Hierarchical policy classes have been additional studied by (He et al., 2010), (Hausknecht and Stone, 2016), (Zheng et al., 2016), and (Andreas et al., 2017). Learning with Weaker Feedback. Our work is motivated by efficient learning under weak expert feedback. When we only receive demonstration data at the high level, and must utilize reinforcement learning at the low level, then our setting can be viewed as an instance of learning under weak demonstration feedback. The 257 primary other way to elicit weaker demonstration feedback is with preference-based or gradient-based learning, studied by (Fürnkranz et al., 2012), (Loftin et al., 2016), and (Christiano et al., 2017). 258 Appendix F APPENDIX TO CHAPTER 7 F.1 Variational Inference Derivation for Hidden Markov Models In this section, we provide the mathematical derivation for the structured variational inference procedure. We focus on the training for Bayesian Hidden Markov Model, in particular the Forward-Backward procedure to complete the description of Algorithm 13. The mathematical details for other types of graphical models depend on the family of such models and should follow similar derivations. Further relevant details on stochastic variational inference can be found in (Hoffman et al., 2013; Johnson and Willsky, 2014; Beal, 2003). d Settings. Given an arbitrarily ordered set of trajectories 𝑈 “ t𝑈1 , . . . , 𝑈𝐾 , 𝐶u, let the coordination mechanism underlying each such 𝑈 be governed by a true unknown model 𝑝, with global parameters 𝜃. We suppress the agent/policy subscript and consider a generic featurized trajectory 𝑥𝑡 “ r𝑢 𝑡 , 𝑐 𝑡 s @𝑡. Let the latent role sequence for the same agent be 𝑧 “ 𝑧 1:𝑇 . ¯ At any time 𝑡, each agent is acting according to a latent role 𝑧𝑡 „ Categoricalt1̄, 2̄, . . . , 𝐾u, which are the local parameters to the structured model. Ideally, role and index asignment can be obtained by calculating the posterior 𝑝p𝑧|𝑥, 𝜃q, which is often intractable. One way to infer the role assignment is via approximating the intractable posterior 𝑝p𝑧|𝑥, 𝜃q using Bayesian inference, typically via MCMC or mean-field variational methods. Since sampling-based MCMC methods are often slow, we instead aim to learn to approximate 𝑝p𝑧|𝑥, 𝜃q by a simpler distribution 𝑞 via Bayesian inference. In particular, we employ techniques from stochastic variational inference (Hoffman et al., 2013), which allows for efficient stochastic training on mini-batches that can naturally integrate with our imitation learning subroutine. Structured Variational Inference for Unsupervised Role Learning. Consider a full probabilistic model: 𝑇 ź 𝑝p𝜃, 𝑧, 𝑥q “ 𝑝p𝜃q 𝑝p𝑧𝑡 |𝜃q𝑝p𝑥𝑡 |𝑧𝑡 , 𝜃q 𝑡 “1 259 with global latent variables 𝜃, local latent variables 𝑧 “ t𝑧𝑡 u𝑇𝑡“1 . Posterior approximation is often cast as optimizing over a simpler model class Q, via searching for global parameters 𝜃 and local latent variables 𝑧 that maximize the evidence lower bound (ELBO) L: log 𝑝p𝑥q ě E𝑞 rlog 𝑝p𝑧, 𝜃, 𝑥qs ´ E𝑞 rlog 𝑞p𝑧, 𝜃qs fi L p𝑞p𝑧, 𝜃qq . Maximizing L is equivalent to finding 𝑞 P Q to minimize the KL divergence KL p𝑞p𝑧, 𝜃|𝑥q||𝑝p𝑧, 𝜃|𝑥qq. For unsupervised structured prediction problem over a family of graphical model, we focus on the structured mean-field variational family, which factorizes 𝑞 as 𝑞p𝑧, 𝜃q “ 𝑞p𝑧q𝑞p𝜃q (Hoffman and Blei, 2014) and decomposes the ELBO objective: L “ E𝑞 rlog 𝑝p𝜃s ´ E𝑞 rlog 𝑞p𝜃s ` E𝑞 rlogp𝑝p𝑧, 𝑥|𝜃qs ´ E𝑞 rlogp𝑞p𝑧qqs. (F.1) This factorization breaks the dependency between 𝜃 and 𝑧, but not between single latent states 𝑧𝑡 , unlike variational inference for i.i.d data (Kingma and Welling, 2013). Variational inference optimizes the objective L typically using natural gradient ascent over global factors 𝑞p𝜃q and local factors 𝑞p𝑧q. (Under mean-field assumption, optimization typically proceeds via alternating updates of 𝜃 and 𝑧.) Stochastic variational inference performs such updates efficiently in mini-batches. For graphical models, structured stochastic variational inference optimizes L using natural gradient ascent over global factors 𝑞p𝜃q and message-passing scheme over local factors 𝑞p𝑧q. We assume the prior 𝑝p𝜃q and complete conditionals 𝑝p𝑧𝑡 , 𝑥𝑡 |𝜃q are conjugate pairs of exponential family, which gives natural gradient of L with respect to 𝑞p𝜃q convenient forms (Johnson and Willsky, 2014). Denote the exponential family forms of 𝑝p𝜃q and 𝑝p𝑧𝑡 , 𝑦 𝑡 |𝜃q by: ln 𝑝p𝜃q “ x𝜂𝜃 , 𝑡 𝜃 p𝜃qy ´ 𝐴𝜃 p𝜂𝜃 q ln 𝑝p𝑧𝑡 , 𝑥𝑡 |𝜃q “ x𝜂 𝑧𝑥 p𝜃q, 𝑡 𝑧𝑥 p𝑧𝑡 , 𝑥𝑡 qy ´ 𝐴𝑧𝑥 p𝜂 𝑧𝑥 p𝜃qq where 𝜂𝜃 and 𝜂 𝑧𝑥 are functions indicating natural parameters, 𝑡 𝜃 and 𝑡 𝑧𝑥 are sufficient statistics and 𝐴p¨q are log-normalizers ((Blei et al., 2017)). Note that in general, different subscripts corresponding to 𝜂, 𝑡, 𝐴 indicate different function parameterization (not simply a change in variable value assignment). Conjugacy in 260 the exponential family yields that (Blei et al., 2017): 𝑡 𝜃 p𝜃q “ r𝜂 𝑧𝑥 p𝜃q, ´𝐴𝑧𝑥 p𝜂 𝑧𝑥 p𝜃qqs and that 𝑝p𝜃|𝑧𝑡 , 𝑥𝑡 q9 exptx𝜂𝜃 ` r𝑡 𝑧𝑥 p𝑧𝑡 , 𝑥𝑡 q, 1s , 𝑡 𝜃 p𝜃qyu (F.2) Conjugacy in the exponential family also implies that the optimal 𝑞p𝜃q is in the same family (Blei et al., 2017), i.e. 𝑞p𝜃q “ exptxr 𝜂𝜃 , 𝑡 𝜃 p𝜃qy ´ 𝐴𝜃 pr 𝜂𝜃 qu for some natural parameters 𝜂r𝜃 of 𝑞p𝜃q. To optimize over global parameters 𝑞p𝜃q, conjugacy in the exponential family allows obtaining convenient expression for the gradient of L with respect to natural parameters 𝜂r𝜃 . The derivation is shown similarly to (Johnson and Willsky, 2014) and (Blei et al., 2017) - we use simplified notations 𝜂r fi 𝜂r𝜃 , 𝜂 fi 𝜂𝜃 , 𝐴 fi 𝐴𝜃 , and ř 𝑡p𝑧, 𝑥q fi 𝑇𝑡“1 r𝑡 𝑧𝑥 p𝑧𝑡 , 𝑥𝑡 q, 1s. Taking advantage of the exponential family identity E𝑞p𝜃 q r𝑡 𝜃 p𝜃qs “ ∇𝐴pr 𝜂q, the objective L can be re-written as: L “ E𝑞p𝜃 q𝑞p𝑧q rln 𝑝p𝜃|𝑧, 𝑥q ´ ln 𝑞p𝜃qs “ x𝜂 ` E𝑞p𝑧q r𝑡p𝑧, 𝑥qs, ∇𝐴pr 𝜂qy ´ pxr 𝜂, ∇𝐴pr 𝜂qy ´ 𝐴pr 𝜂qq Differentiating with respect to 𝜂r, we have that ` ˘` ˘ ∇𝜂rL “ ∇2 𝐴pr 𝜂q 𝜂 ` E𝑞p𝑧q r𝑡p𝑧, 𝑥qs ´ 𝜂r ` ˘´1 r 𝜂r, is defined as ∇ r 𝜂r fi ∇2 𝐴pr The natural gradient of L, denoted ∇ 𝜂q ∇𝜂r. And so the natural gradient of L can be compactly described as: r 𝜂rL “ 𝜂 ` ∇ 𝑇 ÿ E𝑞p𝑧𝑡 q tr𝑡 𝑧𝑥 p𝑧𝑡 , 𝑥𝑡 q, 1su ´ 𝜂r (F.3) 𝑡 “1 Thus a stochastic natural descent update on the global parameters 𝜂r𝜃 proceeds at step 𝑛 by sampling a mini-batch 𝑥𝑡 and taking the global update with step size 𝜌𝑛 : 𝜂r𝜃 Ð p1 ´ 𝜌𝑛 qr 𝜂𝜃 ` 𝜌𝑛 p𝜂𝜃 ` 𝑏 J E𝑞˚ p𝑧𝑡 q r𝑡p𝑧𝑡 , 𝑥𝑡 qsq (F.4) where 𝑏 is a vector of scaling factors adjusting for the relative size of the minibatches. Here the global update assumes optimal local update 𝑞 ˚ p𝑧q has been computed. In each step however, the local factors 𝑞 ˚ p𝑧𝑡 q are computed with mean 261 field updates and the current value of 𝑞p𝜃q (analogous to coordinate ascent). In what follows, we provide the derivation for the update rules for Hidden Markov Models, which are the particular instantiation of the graphical model we use to represent the role transition for our multi-agent settings. Variational factor updates via message passing for Hidden Markov Models. For HMMs, we can view global parameters 𝜃 as the parameters of the underlying HMMs such as transition matrix and emission probabilities, while local parameters 𝑧 govern hidden state assignment at each time step. Fixing the global parameters, the local updates are based on message passing over the graphical model. The exact mathematical derivation depends on the specific graph structure. The simplest scenario is to assume independence among 𝑧𝑡 ’s, which resembles naive Bayes. We instead focus on Hidden Markov Models to capture first-order dependencies in role transitions over play sequences. In this case, “ ‰𝐾 global parameters 𝜃 “ p𝑝 0 , 𝑃, 𝜙q where 𝑃 “ 𝑃𝑖 𝑗 𝑖, 𝑗 “1 is the transition matrix with 𝑃𝑖 𝑗 “ 𝑝p𝑧𝑡 “ 𝑗|𝑧𝑡 ´1 “ 𝑖q, 𝜙 “ t𝜙𝑖 u𝑖𝐾“1 are the emission parameters, and 𝑝 0 is the initial distribution. Consider a Bayesian HMM on 𝐾 latent states. Priors on the model parameters include the initial state distribution 𝑝 0 , transition matrix 𝑃 with rows denoted 𝑝 1 , . . . , 𝑝 𝐾 , and the emission parameters 𝜙 “ t𝜙𝑖 u𝑖𝐾“1 . In this case we have the global parameters 𝜃 “ p𝑝 0 , 𝑃, 𝜙q. For Hidden Markov Model with observation 𝑥 1:𝑇 and latent sequence 𝑧 1:𝑇 , the generative model over the parameters is given by 𝜙𝑖 „ 𝑝p𝜙q (i.i.d from prior), 𝑝𝑖 „ Dirp𝛼𝑖 q, 𝑧1 „ 𝑝 0 , 𝑧𝑡 `1 „ 𝑝 𝑧𝑡 , and 𝑥𝑡 „ 𝑝p𝑥𝑡 |𝜙 𝑧𝑡 q (conditional distribution given parameters 𝜙). We can also write the transition matrix: » fi 𝑝1 — . ffi — 𝑃 “ – .. ffi fl 𝑝𝐾 The Bayesian hierarchical model over the parameters, hidden state sequence 𝑧1:𝑇 , and observation sequence 𝑦 1:𝑇 is iid 𝜙𝑖 „ 𝑝p𝜙q, 𝑝𝑖 „ Dirp𝛼𝑖 q 𝑧1 „ 𝑝 0 , 𝑧𝑡 `1 „ 𝑝 𝑧𝑡 , 𝑥𝑡 „ 𝑝p𝑥𝑡 |𝜙 𝑧𝑡 q ś For HMMs, we have a full probabilistic model: 𝑝p𝑧, 𝑥|𝜃q “ 𝑝 0 p𝑧1 q 𝑇𝑡“1 𝑝p𝑧𝑡 |𝑧𝑡 ´1 , 𝑃q𝑝p𝑥𝑡 |𝑧𝑡 , 𝜙q. Define the likelihood potential 𝐿 𝑡,𝑖 “ 𝑝p𝑥𝑡 |𝜙𝑖 q, the likelihood of the latent sequence, 262 given observation and model parameters, is as follows: 𝑝p𝑧1:𝑇 |𝑥 1:𝑇 , 𝑃, 𝜙q “ ˜ 𝑇 ÿ exp log 𝑝 0 p𝑧1 q ` ¸ 𝑇 ÿ log 𝑃𝑧𝑡 ´1 ,𝑧𝑡 ` 𝑡 “2 log 𝐿 𝑡,𝑧𝑡 ´ 𝑍 (F.5) 𝑡 “1 where 𝑍 is the normalizing constant. Following the notation and derivation from (Johnson and Willsky, 2014), we denote 𝑝p𝑧1:𝑇 |𝑥1:𝑇 ,𝑃,𝜙 q “ HMMp𝑝 0 , 𝑃, 𝐿q. Under mean field assumption, we approximate the true posterior 𝑝p𝑃, 𝜙, 𝑧 1:𝑇 |𝑥 1:𝑇 q with a mean field variational family 𝑞p𝑃q𝑞p𝜙q𝑞p𝑧 1:𝑇 q and update each variational factor in turn while fixing the others. Fixing the global parameters 𝜃, taking expectation of log of (F.5), we derive the r 𝑝r0 , 𝐿q r where: update rule for 𝑞p𝑧q as 𝑞p𝑧1:𝑇 q “ HMMp𝑃, 𝑃r𝑗,𝑘 “ exptE𝑞p𝑃q lnp𝑃 𝑗,𝑘 qu 𝑝r0,𝑘 “ exptln E𝑞p 𝑝0 q 𝑝 0,𝑘 u r 𝑡,𝑘 “ exptE𝑞p𝜙 q lnp𝑝p𝑥𝑡 |𝑧𝑡 “ 𝑘qqu 𝐿 𝑘 To calculate the expectation with respect to 𝑞p𝑧1:𝑇 q, which is necessary for updating other factors, the Forward-Backward recursion of HMMs is defined by forward messages 𝐹 and backward messages 𝐵: 𝐾 ÿ 𝐹𝑡,𝑖 “ r 𝑡,𝑖 𝐹𝑡 ´1, 𝑗 𝑃r𝑗,𝑖 𝐿 (F.6) r 𝑡 `1, 𝑗 𝐵𝑡 `1, 𝑗 𝑃r𝑖, 𝑗 𝐿 (F.7) 𝑗 “1 𝐾 ÿ 𝐵𝑡,𝑖 “ 𝑗 “1 𝐹1,𝑖 “ 𝑝 0 p𝑖q 𝐵𝑇,𝑖 “ 1 As a summary, calculating the gradient w.r.t 𝑧 yields the following optimal variational distribution over the latent sequence: 𝑇 ´ ÿ 𝑞 p𝑧q9 exp E𝑞p𝑃q rln 𝑝 0 p𝑧1 qs ` E𝑞p𝑃q rlog 𝑃𝑧𝑡 ´1 ,𝑧𝑡 s ˚ 𝑡 “2 𝑇 ÿ ` 𝑡 “1 ¯ E𝑞p𝜙q lnr𝑝p𝑥𝑡 |𝑧𝑡 qs , (F.8) 263 which gives the local updates for 𝑞 ˚ p𝑧q, given current estimates of 𝑃 and 𝜙: “ ‰ 𝑃r𝑗,𝑘 “ exp E𝑞p𝑃q lnp𝑃 𝑗,𝑘 q “ ‰ 𝑝rp𝑥𝑡 |𝑧𝑡 “ 𝑘q “ exp 𝐸 𝑞p𝜙q ln 𝑝p𝑥𝑡 |𝑥𝑡 “ 𝑘q , (F.9) (F.10) r 𝑝r to run the forward-backward for 𝑘 “ 1, . . . , 𝐾, 𝑡 “ 1, . . . , 𝑇, and then use 𝑝 0 , 𝑃, algorithm to compute the update 𝑞 ˚ p𝑧𝑡 “ 𝑘q and 𝑞 ˚ p𝑧𝑡 ´1 “ 𝑗, 𝑧𝑡 “ 𝑘q. The forward-backward algorithm in the local update step takes 𝑂p𝐾 2𝑇q time for a chain of length 𝑇 and 𝐾 hidden states. Training to learn model parameters for HMMs. Combining natural gradient step with message-passing scheme for HMMs yield specific update rules for learning the model parameters. Again for HMMs, the global parameters are 𝜃 “ p𝑝 0 , 𝑃, 𝜙q and local variables 𝑧 “ 𝑧1:𝑇 . Assuming the priors on observation parameter 𝑝p𝜙𝑖 q and likelihoods 𝑝p𝑥𝑡 |𝜙𝑖 q are conjugate pairs of exponential family distribution for all 𝑖, the conditionals 𝑝p𝜙𝑖 |𝑥q have the form as seen from equation F.2: 𝑝p𝜙𝑖 |𝑥q9 exptx𝜂 𝜙𝑖 ` r𝑡 𝑥,𝑖 p𝑥q, 1s, 𝑡 𝜙𝑖 p𝜙𝑖 qyu For structured mean field inference, the approximation 𝑞p𝜃q factorizes as 𝑞p𝑃q𝑞p𝑝 0 q𝑞p𝜙q. At each iteration, stochastic variational inference sample a sequence 𝑥1:𝑇 from the data set (e.g. trajectory from any randomly sampled player) and perform stochastic gradient step on 𝑞p𝑃q𝑞p𝑝 0 q𝑞p𝜙q. In order to compute the gradient, we need to calculate expected sufficient statistics w.r.t the optimal factor for 𝑞p𝑧1:𝑇 q, which in turns depends on current value of 𝑞p𝑃q𝑞p𝑝 0 q𝑞p𝜙q. Following the notation from (Johnson and Willsky, 2014), we write the prior and mean field factors as 𝑝p𝑝𝑖 q “ Dirp𝛼𝑖 q, 𝑝p𝜙𝑖 q9 exptx𝜂 𝜙𝑖 , 𝑡 𝜙𝑖 p𝜙𝑖 qyu 𝑞p𝑝𝑖 q “ Dirpr 𝛼𝑖 q, 𝑞p𝜙𝑖 q9 exptxr 𝜂 𝜙𝑖 , 𝑡 𝜙𝑖 p𝜙𝑖 qyu Using message passing scheme as per equations (F.6) and (F.7), we define the 264 intermediate quantities: 𝑇 ÿ p 𝑡𝑥,𝑖 fi E𝑞p𝑧1:𝑇 q Ir𝑧𝑡 “ 𝑖s𝑡𝑥,𝑖 p𝑥𝑡 q 𝑡 “1 𝑇 ÿ “ 𝐹𝑡,𝑖 𝐵𝑡,𝑖 r𝑡𝑥,𝑖 p𝑥𝑡 q, 1s{𝑍 (F.11) 𝑡 “1 𝑇ÿ ´1 pp 𝑡𝑡𝑟𝑎𝑛𝑠,𝑖 q 𝑗 fi E𝑞p𝑧1:𝑇 q Ir𝑧𝑡 “ 𝑖, 𝑧𝑡 `1 “ 𝑗s 𝑡 “1 𝑇ÿ ´1 “ r 𝑡 `1, 𝑗 𝐵𝑡 `1, 𝑗 {𝑍 𝐹𝑡,𝑖 𝑃r𝑖, 𝑗 𝐿 (F.12) 𝑡 “1 pp 𝑡𝑖𝑛𝑖𝑡 q𝑖 fi E𝑞p𝑧1:𝑇 q Ir𝑧1 “ 𝑖s “ 𝑝r0 𝐵1,𝑖 {𝑍 where 𝑍 fi (F.13) ř𝐾 𝑖 “1 𝐹𝑇,𝑖 is the normalizing constant, and I is the indicator function. Given these expected sufficient statistics, the specific update rules corresponding to the natural gradient step in the natural parameters of 𝑞p𝑃q, 𝑞p𝑝 0 q, and 𝑞p𝜙q become: 𝜂r𝜙,𝑖 Ð p1 ´ 𝜌qr 𝜂 𝜙,𝑖 ` 𝜌p𝜂 𝜙,𝑖 ` 𝑏 Jp 𝑡𝑥,𝑖 q (F.14) r𝑖 Ð p1 ´ 𝜌qr 𝛼 𝛼𝑖 ` 𝜌p𝛼𝑖 ` 𝑏 Jp 𝑡𝑡𝑟𝑎𝑛𝑠,𝑖 q (F.15) r0 Ð p1 ´ 𝜌qr 𝛼 𝛼0 ` 𝜌p𝛼0 ` 𝑏 Jp 𝑡𝑖𝑛𝑖𝑡,𝑖 q (F.16) Algorithm 20 Coordinated Structure Learning: LearnStructure t𝑈1 , . . . , 𝑈𝐾 , 𝐶, 𝜃, 𝜌u ÞÑ 𝑞p𝜃, 𝑧q Input: Set of trajectories 𝑈 “ t𝑈 𝑘 u𝐾𝑘“1 . Context 𝐶 Previous parameters 𝜃 “ p𝑝 0 , 𝜃 𝑃 , 𝜃 𝜙 q, stepsize 𝜌 1: 𝑋 𝑘 “ t𝑥 𝑡,𝑘 u𝑇𝑡“1 “ tr𝑢 𝑡,𝑘 , 𝑐 𝑡 su @𝑡, 𝑘.𝑋 “ t𝑋 𝑘 u𝐾 𝑘 “1 r and 𝑝r per equation F.9 and F.10 2: Local update: Compute 𝑃 r 𝑝rq and compute 𝑞p𝑧q “ Forward-Backwardp𝑋, 𝑃, 3: Global update of 𝜃, per equations F.14, F.15, and F.16. output Updated model 𝑞p𝜃, 𝑧q “ 𝑞p𝜃q𝑞p𝑧q F.2 Experimental Evaluation Batch-Version of Algorithm 12 for Predator-Prey Visualizing Role Assignment for Soccer The Gaussian components of latent structure in figure 7.7 give interesting insight about the latent structure of the demonstration data, which correspond to a popular formation arrangement in professional soccer. Unlike the predator-prey domain, 265 Algorithm 21 Multi-Agent Data Aggregation Imitation Learnp𝐴1 , 𝐴2 , . . . , 𝐴𝐾 , 𝐶|𝐷q Input: Ordered actions 𝐴 𝑘 “ t𝑎 𝑡,𝑘 u𝑇𝑡“1 @𝑘, context t𝑐 𝑡 u𝑇𝑡“1 Input: Aggregating data set 𝐷 1 , .., 𝐷 𝐾 for each policy Input: base routine Trainp𝑆, 𝐴q mapping state to actions 1: for 𝑡 “ 0, 1, 2, . . . , 𝑇 do 2: Roll-out 𝑎ˆ 𝑡 `1,𝑘 “ 𝜋 𝑘 p𝑠ˆ𝑡,𝑘 q @ agent 𝑘 3: Cross-update for each policy 𝑘 P t1, . . . , 𝐾u 𝑠ˆ𝑡 `1,𝑘 “ 𝜑 𝑘 pr𝑎ˆ 𝑡 `1,1 , . . . , 𝑎ˆ 𝑡 `1,𝑘 , . . . , 𝑎ˆ 𝑡 `1,𝐾 , 𝑐 𝑡 `1 sq 4: Collect expert action 𝑎 𝑡˚`1,𝑘 given state 𝑠ˆ𝑡 `1,𝑘 @𝑘 5: Aggregate data set 𝐷 𝑘 “ 𝐷 𝑘 Y t𝑠ˆ𝑡 `1,𝑘 , 𝑎 𝑡˚`1,𝑘 u𝑇𝑡 “´01 6: end for 7: 𝜋 𝑘 Ð Trainp𝐷 𝑘 q output 𝐾 new policies 𝜋1 , 𝜋2 , . . . , 𝜋 𝐾 Learning: however, the players are sometimes expected to switch and swap roles. Figure F.1 displays the tendency that each learning policy 𝑘 would takes on other roles outside of its dominant mode. Policies indexed 0 ´ 3 tend to stay most consistent with the Figure F.1: Role frequency assigned to policy, according to the maximum likelihood estimate of the latent structured model prescribed latent roles. We observe that these also correspond to players with the least variance in their action trajectories. Imitation loss is generally higher for less consistent roles (e.g. policies indexed 8 ´ 9). Intuitively, entropy regularization encourages a decomposition of roles that result in learning policies as decoupled as possible, in order to minimize the imitation loss.