Learned Feedback & Feedforward Perception & Control Citation Marino, Joseph Louis (2021) Learned Feedback & Feedforward Perception & Control. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/4mjd-ce53. https://resolver.caltech.edu/CaltechTHESIS:05272021-042158260 Abstract The notions of feedback and feedforward information processing gained prominence under cybernetics, an early movement at the dawn of computer science and theoretical neuroscience. Negative feedback processing corrects errors, whereas feedforward processing makes predictions, thereby preemptively reducing errors. A key insight of cybernetics was that such processes can be applied to both perception, or state estimation, and control, or action selection. The remnants of this insight are found in many modern areas, including predictive coding in neuroscience and deep latent variable models in machine learning. This thesis draws on feedback and feedforward ideas developed within predictive coding, adapting them to improve machine learning techniques for perception (Part II) and control (Part III). Upon establishing these conceptual connections, in Part IV, we traverse this bridge, from machine learning back to neuroscience, arriving at new perspectives on the correspondences between these fields. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Neuroscience, Machine Learning Degree Grantor: California Institute of Technology Division: Biology and Biological Engineering Major Option: Computation and Neural Systems Thesis Availability: Public (worldwide access) Research Advisor(s): Yue, Yisong (co-advisor) Perona, Pietro (co-advisor) Thesis Committee: Yue, Yisong Perona, Pietro O'Doherty, John P. Tsao, Doris Y. (chair) Rao, Rajesh P. N. Defense Date: 21 May 2021 Record Number: CaltechTHESIS:05272021-042158260 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:05272021-042158260 DOI: 10.7907/4mjd-ce53 Related URLs: URL URL Type Description http://proceedings.mlr.press/v80/marino18a.html Publisher Article adapted for chapter 3 http://papers.nips.cc/paper/8011-a-general-method-for-amortizing-variational-filtering Publisher Article adapted for chapter 4 https://openreview.net/forum?id=SyeumQYUUH Publisher Article adapted for chapter 8 http://proceedings.mlr.press/v118/marino20a.html Publisher Article adapted for chapter 5 https://arxiv.org/abs/2010.10670 arXiv Article adapted for chapter 6 https://invertibleworkshop.github.io/INNF_2020/accepted_papers/pdfs/37.pdf Other Article adapted for chapter 7 https://drive.google.com/file/d/10hB0AS_naGNgSNG8p1kqElnc9HRmMYde/view?usp=drivesdk Other Article adapted for chapter 6 https://drive.google.com/open?id=1uc1FdjHVkkEAaU6jz7aVrP3khGyqkdKi Other Article adapted for chapter 9 https://openreview.net/pdf?id=TK_6nNb_C7q Related Document Article extends chapter 5 https://arxiv.org/abs/2011.07464 arXiv arXiv version of chapter 8 ORCID: Author ORCID Marino, Joseph Louis 0000-0001-6387-8062 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 14178 Collection: CaltechTHESIS Deposited By: Joseph Marino Deposited On: 07 Jun 2021 15:47 Last Modified: 08 Nov 2023 00:44 Thesis Files PDF - Final Version See Usage Policy. 17MB Repository Staff Only: item control page

Learned Feedback & Feedforward Perception & Control Thesis by Joseph L. Marino In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2021 Defended May 21, 2021 ii © 2021 Joseph L. Marino ORCID: 0000-0001-6387-8062 All rights reserved iii To my father, Steve, and my grandmother, Elizabeth. iv ACKNOWLEDGEMENTS To the late Jerry Pine, thank you for allowing me into the Caltech community. To my advisors, Yisong Yue and Pietro Perona, thank you for your guidance, encouragement, and compassion. To my collaborators, thank your for your dedication and insights. To the vision lab, Steve Branson, Bo Chen, Ron Appel, Eyrun Eyjolfsdottir, Krzysztof Chalupka, David Hall, Mason McGill, Matteo Ruggero Ronchi, Grant Van Horn, Serim Ryou, Alvita Tran, Oisin Mac Aodha, Cristina Segalin, Tony Zhang, Sara Beery, Eli Cole, Jennifer Sun, and Neehar Kondapaneni: thank you for all of the wonderful memories. To the Yue Crew, thank you for broadening my perspectives. Special thanks to my fellow former physicists Jeremy Bernstein and Uriah Israel. To my CNS cohort, Yang Liu and Koichiro Kajikawa, thank you for your friendship. To IMPLiCIT, thank you for the improvisational space to explore ideas. To all of the other friends, thank you for making my time at Caltech memorable. To the Hairy Chests, Matteo Ruggero Ronchi and Alessandro Zocca, thank you for the adventures and support. Our Italian coffee breaks were a highlight of the day. To my family, thank you for your love. v ABSTRACT The notions of feedback and feedforward information processing gained prominence under cybernetics, an early movement at the dawn of computer science and theoretical neuroscience. Negative feedback processing corrects errors, whereas feedforward processing makes predictions, thereby preemptively reducing errors. A key insight of cybernetics was that such processes can be applied to both perception, or state estimation, and control, or action selection. The remnants of this insight are found in many modern areas, including predictive coding in neuroscience and deep latent variable models in machine learning. This thesis draws on feedback and feedforward ideas developed within predictive coding, adapting them to improve machine learning techniques for perception (Part II) and control (Part III). Upon establishing these conceptual connections, in Part IV, we traverse this bridge, from machine learning back to neuroscience, arriving at new perspectives on the correspondences between these fields. vi PUBLISHED CONTENT AND CONTRIBUTIONS Yang, Ruihan, Yibo Yang, Joseph Marino, and Stephan Mandt (2021). “Hierarchical Autoregressive Modeling for Neural Video Compression”. In: International Conference on Learning Representations. url: https://openreview.net/ pdf?id=TK_6nNb_C7q. J.M. participated in the conception of the project and the writing of the manuscript. Guerra, Alex and Joseph Marino (2020). “Sequential Autoregressive Flow-Based Policies”. In: ICML workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models. J.M. participated in the conception of the project, conducting experiments, and the writing of the manuscript. Marino, Joseph, Lei Chen, Jiawei He, and Stephan Mandt (2020). “Improving Sequential Latent Variable Models with Autoregressive Flows”. In: Symposium on Advances in Approximate Bayesian Inference, pp. 1–16. url: http : / / proceedings.mlr.press/v118/marino20a.html. J.M. participated in the conception of the project, implementation, analysis, and the writing of the manuscript. Marino, Joseph, Alexandre Piché, Alessandro Davide Ialongo, and Yisong Yue (2020). “Iterative Amortized Policy Optimization”. In: Preprint. url: https: //arxiv.org/abs/2010.10670. J.M. participated in the conception of the project, conducting experiments, and the writing of the manuscript. He, Jiawei, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann (2019). “Variational autoencoders with jointly optimized latent dependency structure”. In: International Conference on Learning Representations. url: https : / / openreview.net/pdf?id=SJgsCjCqt7. J.M. participated in the conception of the project, derivations, analysis, and writing of the manuscript. Marino, Joseph (2019). “Predictive Coding, Variational Autoencoders, and Biological Connections”. In: NeurIPS Workshop on Real Neurons and Hidden Units. url: https://openreview.net/forum?id=SyeumQYUUH. J.M. participated in the conception of the project and the writing of the manuscript. Marino, Joseph, Alexandre Piché, and Yisong Yue (2019). “On the Design of Variational RL Algorithms”. In: NeurIPS Workshop on Deep Reinforcement Learning. url: https://drive.google.com/file/d/10hB0AS_naGNgSNG8p1kqElnc9HRmMYde/ view?usp=drivesdk. J.M. participated in the conception of the project, conducting experiments, and the writing of the manuscript. vii He, Jiawei, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal (2018). “Probabilistic video generation using holistic attribute control”. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 452–467. url: http://openaccess.thecvf.com/content_ECCV_2018/html/Jiawei_ He_Probabilistic_Video_Generation_ECCV_2018_paper.html. J.M. participated in the conception of the project, derivations, baseline comparisons, and the writing of the manuscript. Marino, Joseph, Milan Cvitkovic, and Yisong Yue (2018). “A general method for amortizing variational filtering”. In: Advances in Neural Information Processing Systems, pp. 7857–7868. url: http://papers.nips.cc/paper/8011- ageneral-method-for-amortizing-variational-filtering. J.M. participated in the conception of the project, implementation, analysis, and the writing of the manuscript. Marino, Joseph, Yisong Yue, and Stephan Mandt (2018). “Iterative Amortized Inference”. In: International Conference on Machine Learning, pp. 3403–3412. url: http://proceedings.mlr.press/v80/marino18a.html. J.M. participated in the conception of the project, implementation, analysis, and the writing of the manuscript. viii TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Published Content and Contributions . . . . . . . . . . . . . . . . . . . . . . vi Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi I Introduction 1 Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Cybernetics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Neuroscience and Machine Learning, Convergence and Divergence . 1.3 Predictive Coding Meets Machine Learning . . . . . . . . . . . . . . 1.4 The Future of Feedback & Feedforward . . . . . . . . . . . . . . . . Chapter II: Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Probabilistic Models . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Variational Inference . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 3 7 8 15 15 15 23 30 II Perception 37 Chapter III: Iterative Amortized Inference . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Issues with Direct Inference Models . . . . . . . . . . . . . . . . . . 3.3 Iterative Amortized Inference . . . . . . . . . . . . . . . . . . . . . 3.4 Iterative Inference in Latent Gaussian Models . . . . . . . . . . . . . 3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter IV: Amortized Variational Filtering . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Variational Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Appendix: Filtering ELBO Derivation . . . . . . . . . . . . . . . . Chapter V: Improving Sequential Latent Variable Models with Autoregressive Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 38 39 42 44 47 52 56 56 57 60 64 70 70 76 76 ix 5.2 5.3 5.4 5.5 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix: ELBO Derivation . . . . . . . . . . . . . . . . . . . . . IIIControl 77 80 86 86 90 Chapter VI: Iterative Amortized Policy Optimization . . . . . . . . . . . . . 91 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.3 Iterative Amortized Policy Optimization . . . . . . . . . . . . . . . 98 6.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Chapter VII: Sequential Autoregressive Flow-Based Policies . . . . . . . . . 114 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 7.2 Autoregressive Flow-Based Policies . . . . . . . . . . . . . . . . . . 115 7.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 7.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 IV Discussion 124 Chapter VIII: Connections to Predictive Coding & Neuroscience . . . . . . . 125 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.2 Predictive Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.3 Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8.4 Correspondences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Chapter IX: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 9.1 Iterative Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 158 9.2 Combining Generative Perception & Control . . . . . . . . . . . . . 160 9.3 Controlling Perception . . . . . . . . . . . . . . . . . . . . . . . . . 163 9.4 Bridging Neuroscience and Machine Learning . . . . . . . . . . . . 169 x LIST OF ILLUSTRATIONS Number Page 1.1 Conceptual Overview. The concepts put forth by cybernetics permeated the modern fields of theoretical neuroscience (left), machine learning (center), and control theory (right). The graph attempts to identify key concepts and several relevant works in these areas, although the graph can barely begin to capture the full breadth and depth of prior work, as well as the overlap between control theory and theoretical neuroscience. At the bottom of the graph, we have coarsely labeled how Parts II & III of this thesis fit into the conceptual landscape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Maximum Likelihood Estimation. A data distribution, 𝑝 data , produces samples, x, of a random variable 𝑋. Maximum likelihood estimation attempts to fit a model distribution, 𝑝 𝜃 , to the empirical distribution of samples by maximizing the log-likelihood of data samples under the model (Eq. 2.3). . . . . . . . . . . . . . . . . . . 16 2.2 Dependency Structures. Circles denote random variables, with gray denoting observed variables and white denoting latent variables. Arrows denote probabilistic conditional dependencies. From left to right: autoregressive model (Eqs. 2.5 & 2.6), latent variable model (Eq. 2.7), hierarchical latent variable model (Eq. 2.10), autoregressive or sequential latent variable model (Eq. 2.11). For clarity, we have drawn a subset of the possible dependencies in the final model. . . . . 18 2.3 Model Parameterization & Computation Graph. The diagram depicts a simplified computation graph for a deep autoregressive Gaussian model. Green circles denote the conditional likelihood distribution at each step, while gray circles again denote the (distribution of) data observations. Smaller red circles denote each of the log-likelihood terms in the objective. Gradients w.r.t. these terms are backpropagated through the networks parameterizing the model’s distribution parameters (red dotted lines). . . . . . . . . . . . . . . . 22 xi 2.4 ELBO Computation Graphs. (a) Basic computation graph for variational inference. Outlined circles denote distributions. Smaller red circles denote terms in the ELBO objective. Arrows, again, denote conditional dependencies. This notation can be used to express (b) hierarchical and (c) sequential models with various model dependencies. 25 2.5 Stochastic Gradient Estimation. (a) Variational inference with parametric approximate posteriors requires optimizing L w.r.t. the distribution parameters, λ. This often requires estimating stochastic gradients (red dotted lines). (b) The score function estimator (Eq. 2.26) is applicable to any distribution, but suffers from high variance. Note: the solid red arrow denotes the per-sample objective, 𝑙 (x, z; 𝜃). (c) The pathwise derivative estimator (Eq. 2.27), in contrast, has lower variance, but is less widely applicable. . . . . . . . . . . . . . . . . . 27 2.6 Variational Autoencoder (VAE). VAEs combine direct amortization (Eq. 2.28) and the pathwise derivative estimator (Eq. 2.27, top) with Gaussian approximate posteriors to train deep latent variable models. In the model diagram (center), the amortized inference model (left) acts as an encoder, with the conditional likelihood (right) acting as a decoder. Each are parameterized by deep networks. . . . . . . . . . 29 3.1 The Amortization Gap. Optimization surface of L (in nats) for a 2-D latent Gaussian model and an MNIST data example. Shown on the plots are the optimal estimate (F), the output of a direct inference model (), and an optimization trajectory of gradient ascent (•), initialized at the blue square at (0,0). The plot on the right shows an enlarged view near the optimum. Gradient-based optimization outperforms the direct inference model, exhibiting an amortization gap in performance: L (F) > L (•) > L (). . . . . . . . . . . . . . 40 3.2 Lack of Prior Information. Naïve direct inference models, encoding only observations, cannot account for conditional priors in structured latent variable models. These arise in hierarchical (left) and sequential (right) latent variable models. For instance, in hierarchical models, bottom-up approximate posteriors at intermediate levels cannot account for conditional “top-down” priors which result from sampling higher-level latent variables. . . . . . . . . . . . . . . . . . . . . . . 41 xii 3.3 Iterative Amortized Inference. Computation graph for a latent variable model with iterative amortized inference. Red dots, denoting errors or gradients from the ELBO objective, are used to update the current estimate of the approximate posterior parameters, λ. Using the pathwise derivative estimator, one can update the inference model parameters, 𝜙, learning to efficiently perform iterative inference. . . . 3.4 Automatic Top-Down Inference. Bottom-up direct amortized inference cannot account for conditional priors, whereas iterative amortized inference automatically accounts for these priors through gradients or errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Iterative Amortized Inference Optimization. Optimization trajectory on L (in nats) for an iterative inference model with a 2D latent Gaussian model for a particular MNIST example. The estimate is initialized at (0, 0) (cyan square), and the iterative inference model adaptively adjusts inference update step sizes to iteratively refine the approximate posterior estimate () to arrive at the optimum (F). . . . 3.6 Amortized vs. Gradient-Based Optimization. Average ELBO over MNIST validation set during inference optimization as a function of (a) inference iterations and (b) inference wall-clock time. Iterative amortized inference converges faster than gradient-based optimizers to better estimates, remaining stable over hundreds of iterations, despite only being trained with 16 inference iterations. . . . . . . . . . . . . 3.7 Reconstructions Over Inference Iterations. During inference (left to right), reconstruction means become gradually sharper, more closely resembling data examples (right). . . . . . . . . . . . . . . . . . . . 3.8 Reduced Gradient Magnitudes. Gradient magnitudes (vertical axis) over inference iterations (indexed by color on right) during training (horizontal axis) on RCV1. Approximate posterior mean gradient magnitudes decrease over inference iterations as estimates approach local maxima. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Hyperparameter Comparison. ELBO for direct and iterative inference models on binarized MNIST for (a) additional inference iterations during training and (b) additional samples. Iterative inference models improve significantly with both quantities. . . . . . . . . . . . . . . . 43 45 47 49 50 50 51 xiii 4.1 Variational Filtering Inference. The diagram shows filtering inference within a sequential latent variable model, as outlined in Algorithm 2. The central gray region depicts inference optimization at step 𝑡, initialized at or near the corresponding prior (white), 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). Sampling from the approximate posterior (blue) generates the conditional likelihood (green), 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ), which is evaluated at the observation (gray), x𝑡 , to calculate the prediction error. This term is combined with the KL divergence between the approximate posterior and prior, yielding the step ELBO (red), L𝑡 (Eq. 4.7). Inference optimization (E-step) involves finding the approximate posterior that maximizes the step ELBO terms. . . . . . . . . . . . . . . . . . . . 61 4.2 Filtering Inference Models for VRNN, SRNN, SVG, and AVF. Each diagram shows the computational graph for inferring the approximate posterior parameters, λ𝑞 , at step 𝑡. Previously proposed methods rely on hand-crafted architectures of observations, hidden states, and latent variables. AVF is a simple, general filtering procedure that only requires the local inference gradient. . . . . . . . . . . . . . . . . . . 65 4.3 Prediction-Update Visualization. Test data (top), output predictions (middle), and updated reconstructions (bottom) for TIMIT using SRNN with AVF. Sequences run from left to right. The predictions made by the model already contain the general structure of the data. AVF explicitly updates the approximate posterior from the prior prediction, focusing on inference corrections rather than re-estimation. 66 4.4 Improvement with Inference Iterations. Results are shown on the TIMIT validation set using VRNN with AVF. (a) Average negative ELBO per step with varying numbers of inference iterations during training. Additional iterations result in improved performance. (b) Average relative ELBO improvement from the initial (prior) estimate at each inference iteration for a single model. Empirically, each successive iteration provides diminishing additional improvement. . . 68 5.1 Redundancy Reduction. (a) Conditional densities for 𝑝(𝑥 2 |𝑥 1 ). (b) The marginal, 𝑝(𝑥 2 ) differs from the conditional densities, thus, I (𝑥 1 ; 𝑥 2 ) > 0. (c) In the normalized space of 𝑦, the corresponding densities 𝑝(𝑦 2 |𝑦 1 ) are identical. (d) The marginal 𝑝(𝑦 2 ) is identical to the conditionals, so I (𝑦 1 ; 𝑦 2 ) = 0. Thus, in this case, a conditional affine transform removed the dependencies. . . . . . . . . . . . . . . 78 xiv 5.2 Model Diagrams. (a) An autoregressive flow pre-processes a data sequence, x1:𝑇 , to produce a new sequence, y1:𝑇 , with reduced temporal dependencies. This simplifies dynamics modeling for a higher-level sequential latent variable model, 𝑝 𝜃 (y1:𝑇 , z1:𝑇 ). Empty diamond nodes represent deterministic dependencies, not recurrent states. (b) Diagram of the autoregressive flow architecture. Blank white rectangles represent convolutional layers (see Appendix). The three stacks of convolutional layers within the blue region are shared. cat denotes channel-wise concatenation. . . . . . . . . . . . . . . . . . . 5.3 Sequence Modeling with Autoregressive Flows. Top: Pixel values (solid) for a particular pixel location in a video sequence. An autoregressive flow models the pixel sequence using an affine shift (dashed) and scale (shaded), acting as a frame of reference. Middle: Frames of the data sequence (top) and the resulting “noise” (bottom) from applying the shift and scale. The redundant, static background has been largely removed. Bottom: The noise values (solid) are modeled using a base distribution (dashed and shaded) provided by a higher-level model. By removing temporal redundancy from the data sequence, the autoregressive flow simplifies dynamics modeling. . . . 5.4 Flow Visualization for SLVM + 1-AF on Moving MNIST (left) and KTH Actions (right). . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Improved Generated Samples. Random samples generated from (a) VideoFlow and (b) VideoFlow + AF, each conditioned on the first 3 frames. Using AF produces more coherent samples. The robot arm blurs for VideoFlow in samples 1 and 4 (red), but does not blur for VideoFlow + AF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Decreased Temporal Correlation. (a) Affine autoregressive flows result in sequences, y1:𝑇 , with decreased temporal correlation, corry , as compared with that of the original data, corrx . (b) For SLVM + 1-AF, corry decreases during training on KTH Actions. . . . . . . . . 5.7 Improved Generalization. The low-level reference frame improves generalization to unseen sequences. Train and test negative loglikelihood bound histograms for (a) SLVM and (b) SLVM + 1-AF on KTH Actions. (c) The generalization gap for SLVM + 1-AF remains small for varying amounts of KTH training data, while it becomes worse in the low-data regime for SLVM. . . . . . . . . . . . . . . . . 79 81 82 83 84 85 xv 6.1 Amortized Policy Optimization. (a) 2D visualization of policy optimization. A direct amortized policy network fails to output an optimal estimate, resulting in an amortization gap in performance. An iterative amortized policy network finds an improved estimate. (b) Diagrams of direct and iterative amortized policy optimization. Larger circles denote distributions, and smaller red circles denote terms in the objective, J . Dashed arrows denote amortized optimizers. Iterative amortization uses gradient-based feedback during optimization, whereas direct amortization does not. . . . . . . . . . . 95 6.2 Estimating Multiple Policy Modes. Unlike direct amortization, which is restricted to a single estimate, the stochasticity of iterative optimization allows iterative amortization to effectively sample from multiple high-value modes in the action space. This capability is shown for a particular state in Ant-v2, showing multiple optimization runs across two action dimensions (Left). Each colored square denotes an initialization. The optimizer finds both modes, with the assigned density plotted on the Right. This capability provides increased flexibility in action exploration. . . . . . . . . . . . . . . . . . . . . 100 6.3 Mitigating Value Overestimation. Using the same value estimation setup (𝛽 = 1), shown on Ant-v2, iterative policy optimization results in (a) higher value overestimation bias and (b) a more rapidly changing policy as compared with direct policy optimization. Increasing 𝛽 helps to mitigate these issues by further penalizing variance in the value estimate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6.4 Policy Optimization. Visualization over time steps of (a) one dimension of the policy distribution and (b) the improvement in the objective, Δ J , across policy optimization iterations. . . . . . . . . . 102 6.5 Performance Comparison. Iterative amortized policy optimization performs comparably with or better than direct amortized policies across a range of MuJoCo environments. Performance curves show the mean and ± standard deviation over 5 random seeds. . . . . . . . 103 xvi 6.6 Amortization Gap. Estimated amortization gaps per step for direct and iterative amortized policy optimization. Iterative amortization achieves comparable or lower gaps across environments. Gaps are estimated using stochastic gradient-based optimization over 100 random states. Curves show the mean and ± standard deviation over 5 random seeds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.7 Additional Performance Comparison. Results are shown on the remaining MuJoCo environments from OpenAI gym. Performance curves show the mean and ± standard deviation over 5 random seeds. 104 6.8 Multiple Policy Modes. (a) Histogram of distances between policy means (µ) across optimization runs (𝑖 and 𝑗) over seeds and states on Walker2d-v2 at 3 million environment steps. For the state with the largest distance, (b) shows the projected optimization surface on each pair of action dimensions, and (c) shows the policy density for 10 optimization runs. . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.9 Varying Iterations During Training. Performance of iterative amortized policy optimization for varying numbers of iterations during training. Increasing the number of iterations generally results in improvements. Curves show the mean and ± standard deviation over 5 random seeds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.10 Amortization Gap of Varying Iterations During Training. Corresponding amortization gaps for varying numbers of iterations during training. We generally see that increasing the number of iterations generally reduces the amortization gap. . . . . . . . . . . . . . . . . 106 6.11 Comparison with Iterative Optimizers. Average estimated objective over policy optimization iterations, comparing with Adam (Kingma and Ba, 2014) and CEM (Rubinstein and Kroese, 2013). These iterative optimizers require over an order of magnitude more iterations to reach comparable performance with iterative amortization, making them impractical in many applications. . . . . . . . . . . . . 106 6.12 Optimizing Model-Based Value Estimates. (a) Performance comparison of direct and iterative amortization using model-based value estimates. (b) Planned trajectories over policy optimization iterations. (c) The corresponding estimated objective increases over iterations. (d) Zero-shot transfer of iterative amortization from model-free (MF) to model-based (MB) estimates. . . . . . . . . . . . . . . . . . . . . 108 xvii 7.1 Autoregressive Flow-Based Policy. An autoregressive flow-based policy converts samples from a base distribution, 𝜋 𝜙 (u𝑡 |s𝑡 ), using an affine transform with parameters β𝜃 (a<𝑡 ) and δ𝜃 (a<𝑡 ), into actions, a𝑡 . The affine transform (top) acts as a feedforward policy, purely conditioned on previous actions, whereas the base distribution (bottom) acts as a feedback policy. Each are parameterized by a separate deep network, with parameters 𝜃 and 𝜙, respectively. . . . . . . . . . . . . 116 7.2 Performance Comparison, 2×256 (Default) Policy Network. Comparison between SAC and ARSAC with an action window of 5 time steps across dm_control suite environments. Each curve shows the mean and standard deviation across 5 random seeds. . . . . . . . . . 118 7.3 Humanoid Performance Comparison, 2 × 256 (Default) Policy Network. Comparison between SAC and ARSAC with varying action window sizes on humanoid tasks from dm_control suite. Each curve shows the mean and standard deviation across 5 random seeds. 119 7.4 Performance Comparison, 1 × 32 Policy Network. Comparison between SAC and ARSAC with an action window of 5 time steps across dm_control suite environments. Each curve shows the mean and standard deviation across 5 random seeds. . . . . . . . . . . . . 119 7.5 Policy Visualization & Distillation. Left: Visualization of the base distribution and autoregressive flow for various action dimensions across various environments. Right: Visualization after policy distillation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 8.1 Spatiotemporal Predictive Coding. (a) Spatial predictive coding models and removes spatial dependencies. In the domain of natural images, one version of linear predictive coding is ZCA whitening, which yields center-surround filters (left). As a result, the whitened image contains highlighted edges (right). (b) Temporal predictive coding models and removes temporal dependencies. In the domain of natural video, this tends to remove static backgrounds. . . . . . . . . 127 xviii 8.2 Brain Anatomy & Cortical Circuitry. Left: Sensory inputs enter first-order relays in thalamus from sensory organs. Thalamus forms reciprocal connections with neocortex. Neocortex consists of hierarchies of cortical areas, with both forward and backward connections. Right: Neocortex is composed of six layers (I–VI), with specific neuron classes and connections at each layer. The simplified schematic depicts two cortical columns. Black and red circles represent excitatory and inhibitory neurons respectively, with arrows denoting major connections. This basic circuit motif is repeated with slight variations throughout neocortex. . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.3 Hierarchical Predictive Coding. The diagram shows the basic computation graph for a Gaussian latent variable model with MAP inference. The insets show the weighted error calculation for the latent (left) and observed (right) variables. . . . . . . . . . . . . . . 131 8.4 Hierarchical Predictive Coding & VAEs. Computation diagrams for (a) hierarchical predictive coding, (b) VAE with direct amortized inference, and (c) VAE with iterative amortized inference (Chapter 3). J| denotes the transposed Jacobian matrix of the generative model’s conditional likelihood. Red dotted lines denote gradients, and black dashed lines denote amortized inference. Hierarchical predictive coding and VAEs are highly similar in both their model formulation and inference approach. . . . . . . . . . . . . . . . . . . . . . . . . 138 8.5 Pyramidal Neurons & Deep Networks. Connecting deep latent variable models with predictive coding places deep networks (bottom) in correspondence with the dendrites of pyramidal neurons (top). This is in contrast with conventional one-to-one analogies of biological and artificial neurons, suggesting a larger role for non-linear dendritic computation and alternative correspondences for backpropagation. . . 140 8.6 Pyramidal Neurons & Amortization. In predictive coding, inference updating is implemented using forward pyramidal neurons in cortex, taking prediction errors as input. In deep latent variable models, iterative amortized inference plays a similar role, continuing the analogy of pyramidal neurons and deep networks. Interestingly, this suggests a separation of processing in apical and basal dendrites, incorporating errors from the current and lower latent level. . . . . . 141 xix 8.7 Backpropagation. Placing deep networks in correspondence with pyramidal neuron dendrites suggests an alternative perspective on the biological-plausibility of backpropagation. In deep latent variable models, backpropagation is only performed across variables that are directly connected through a conditional probability (left). From the perspective presented here, this corresponds to learning within pyramidal neurons. One possible implementation may be through backpropagating action potentials, perhaps combined with other neuromodulatory inputs (right). . . . . . . . . . . . . . . . . . . . . 142 8.8 Computational Schematic of the Visual Pathway. Interpreting the early visual pathway from the perspective of a latent variable model, we can assign computational functions to retina, LGN, and cortex. Retina and LGN are interpreted as implementing normalizing flows, i.e. spatiotemporal predictive coding, reducing spatial and temporal redundancy in the visual input. LGN also serves as the lowest level for hierarchical predictions, which are computed through backward connections in cortex. Using prediction errors throughout the hierarchy, forward cortical connections update latent estimates. . . 145 8.9 Adding & Removing Dependencies with Normalizing Flows. Normalizing flows provides a general mathematical framework for removing (left) or adding (right) probabilistic dependencies. Using lateral interactions, one can move between a normalized (top) or un-normalized (bottom) space. Normalized spaces have benefits for compression, whereas un-normalized spaces are more expressive. Neural systems may employ these transforms for sensory and motor processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 9.1 Thesis Summary. Graphical representations of perception (a–c) and control (d–e) using feedback (a, b, d) and feedforward (c, e) processes.159 9.2 Combining Generative Perception & Control. Graphical representation combining generative perception (Part II) and control (Part III). Internal perceptual latent variables are a hierarchical variable in the control policy. Importantly, the agent maximizes the task-relevant information gain from the environment’s state observation, rather than the conditional log-likelihood. . . . . . . . . . . . . . . . . . . . . . 162 xx 9.3 State Distributions & Rewards. A quadratic reward function, 𝑟 (𝑆), can be equivalently expressed as a Gaussian desired state distribution, 𝑝 𝑑 (𝑆). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.4 Comparator Circuit. Using the error between the current state and the input reference signal (or setpoint), a compensator-effector function updates the output control. The feedback takeoff function converts this control output into the current state. An ideal compensator-effector minimizes the discrepancy between the current state and the reference signal. Adapted from Wiener, 1948. . . . . . . . . . . . . . . . . . . 167 xxi LIST OF TABLES Number Page 1.1 Thesis Organization. . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.1 Negative Log-Likelihood on binarized MNIST (in nats) for direct and iterative amortized inference. . . . . . . . . . . . . . . . . . . . 52 3.2 Negative Log-Likelihood on CIFAR-10 (in bits/dim.) for direct and iterative amortized inference. . . . . . . . . . . . . . . . . . . . . . 52 3.3 Perplexity on RCV1 for direct and iterative amortized inference. . . . 52 4.1 Average negative ELBO per step (in nats) on the TIMIT speech dataset for SRNN and VRNN with the respective originally proposed filtering procedures (baselines) and with AVF. . . . . . . . . . . . . 69 4.2 Average negative ELBO per step (in nats per dimension) on the KTH Actions video dataset for SVG with the originally proposed filtering procedure (baseline) and with AVF. . . . . . . . . . . . . . . 69 4.3 Average negative ELBO per step (in nats) on polyphonic music datasets for SRNN with and without AVF. Results from Fraccaro, S. K. Sønderby, et al., 2016 are provided for comparison, however, our model implementation differs in several aspects (see Marino, Cvitkovic, and Yue, 2018). . . . . . . . . . . . . . . . . . . . . . . . 69 5.1 Quantitative Comparison. Average test and (train) negative loglikelihood in nats per dimension for Moving MNIST, BAIR Robot Pushing, and KTH Actions. Lower values are better. . . . . . . . . . 85 8.1 Proposed Neural Correspondences of Hierarchical Predictive Coding. 133 Part I Introduction 1 2 Chapter 1 INTRODUCTION This thesis operates at multiple levels. At the lowest level, this thesis presents two core techniques for improving probabilistic modeling, applied to both state estimation, or perception, and action selection, or control. At a higher level, this thesis fortifies a bridge between neuroscience and machine learning through the theory of predictive coding. And at a higher level still, this thesis is an attempt to help re-integrate the concepts and framework of cybernetics back into the current scientific discourse. 1.1 Cybernetics The field of cybernetics (Wiener, 1948; Ashby, 1956) was an agglomeration of various disciplines, encompassing aspects of what would become computer science, neuroscience, control theory, and the social sciences. Despite their diversity, a common theme unified these interdisciplinary investigations: feedback. Broadly, feedback is the process of feeding the output of a system back into the system itself. Such processes come in two flavors, with positive feedback amplifying a signal and negative feedback attenuating a signal. Negative feedback is particularly useful in engineering applications (Astrom and R. M. Murray, 2008), where it can reduce the error between two signals. A related concept is that of feedforward processes, which attempt to predict a signal, preemptively reducing error. These ideas around information processing formed the basis of cybernetics’ formulation of perception and control, both in biological and non-biological systems, in terms of feedback and feedforward processes. Perception and control, though seemingly disparate concepts, are intimately related. In both cases, given a model of how the corresponding variable (states or actions) affects observed outcomes, one can 1) infer an estimate of the variable that would result in an observation, and 2) learn an improved model to better predict observations. Inference and learning, by minimizing prediction errors, are negative feedback processes, while the model itself, formulated across time, may involve feedforward processes. Cybernetics formalized these techniques using probabilistic models, i.e., models that estimate the likelihood of random outcomes, and variational calculus, an optimization technique for estimating functions, including probability distributions (Wiener, 1948). By using these techniques to evaluate and minimize 3 error signals, such systems can exhibit goal-directed, or teleological, behavior, in which desired outcomes appear to drive action. This provided a novel perspective in understanding how goal-directed behavior can arise in biological organisms composed of ordinary matter (Rosenblueth, Wiener, and Bigelow, 1943), as well as a path toward constructing more capable machines. With these new techniques and unifying perspectives, cybernetics helped usher in a new era of interdisciplinary research, aiming to establish common computational principles underlying both biological and non-biological systems. At this intersection came the first computational models of neuron function (McCulloch and Pitts, 1943; Rosenblatt, 1958), a formal definition of information (Wiener, 1942; Shannon, 1948) (with connections to neural systems (Barlow, 1961)), and algorithms for negative feedback perception and control (MacKay, 1956; Kalman, 1960). While further advances continued in these directions (see Prieto et al. (2016) and references therein), the field of cybernetics disbanded due to a variety of conspiring factors (Conway and Siegelman, 2006), with the new techniques and ideas surviving in the offshoots of theoretical neuroscience, machine learning, control theory, etc. Figure 1.1 outlines the progression of a subset of these ideas that are relevant to this thesis. 1.2 Neuroscience and Machine Learning, Convergence and Divergence As the descendant fields of cybernetics progressed in their respective areas, a renewed dialogue formed between neuroscience and machine learning in the 1980s–1990s. Neuroscientists, bolstered by new physiological and functional analyses, began making traction in studying neural systems in probabilistic and information-theoretic terms (Laughlin, 1981; Srinivasan, Laughlin, and Dubs, 1982; Barlow, 1989; Bialek et al., 1991). In machine learning, improvements in probabilistic modeling (Pearl, 1986) and artificial neural networks (Rumelhart, Hinton, and Williams, 1986) combined with ideas from statistical mechanics (Hopfield, 1982; Ackley, Hinton, and Sejnowski, 1985) to yield new classes of models and training techniques. This convergence of ideas, primarily centered around perception, resulted in new theories of neural processing and improvements in their mathematical underpinnings. In particular, the notion of predictive coding emerged within neuroscience (Srinivasan, Laughlin, and Dubs, 1982; Rao and Ballard, 1999). In its most general form, predictive coding postulates that neural circuits are fundamentally engaged in estimating probabilistic models of other neural activity and the surrounding environment, with feedback and feedforward processes playing a central role. These 4 models were initially formulated in early sensory areas, e.g., retina (Srinivasan, Laughlin, and Dubs, 1982) and thalamus (Dong and Atick, 1995), using feedforward processes to predict future neural activity. Similar notions were also extended to higher-level sensory processing in neocortex. In a series of papers by David Mumford (Mumford, 1991; Mumford, 1992), top-down neural projections (from higher-level to lower-level sensory areas) were hypothesized to convey hierarchical sensory predictions, whereas bottom-up neural projections were hypothesized to convey prediction errors. Through a negative feedback process, these errors would then update state estimates. These ideas were formalized and analyzed by Rao and Ballard, 1999, formulating a simplified artificial neural network model of images, reminiscent of a Kalman filter (Kalman, 1960). Feedback and feedforward processes also featured prominently in machine learning. Indeed, the primary training algorithm for artificial neural networks, backpropagation (Rumelhart, Hinton, and Williams, 1986), literally feeds (propagates) the output prediction errors back through the network, i.e., negative feedback. During this period, the technique of variational inference was rediscovered within machine learning (Hinton and Van Camp, 1993; Neal and Hinton, 1998), recasting approximate probabilistic inference using variational calculus. This technique proved essential in formulating the Helmholtz machine (Dayan, Hinton, et al., 1995; Dayan and Hinton, 1996), a hierarchical probabilistic model parameterized by artificial neural networks. Similar advances were made in autoregressive probabilistic models (Frey, Hinton, and Dayan, 1996; Y. Bengio and S. Bengio, 2000), using artificial neural networks to form sequential feedforward predictions, as well as new classes of invertible probabilistic models (Comon, 1994; Parra, Deco, and Miesbach, 1995; Deco and Brauer, 1995; Bell and Sejnowski, 1997). Unfortunately, as the field moved toward simpler, more tractable models in the late 1990s, funding became scarce, and the resulting chill of an “AI winter” slowed progress in these areas. These new ideas regarding variational inference and probabilistic models, particularly the Helmholtz machine (Dayan, Hinton, et al., 1995), influenced predictive coding. Specifically, Karl Friston utilized variational inference to formulate hierarchical dynamical models of neocortex (Friston, 2005; Friston, 2008). In line with Mumford’s proposal (Mumford, 1992), these models contain multiple levels of variables, with each level attempting to predict its future activity (feedforward) as well as the activity at lower levels, closer to the input data. Through variational inference, prediction errors across levels facilitate updating higher-level estimates (negative 5 Cybernetics <latexit sha1_base64="xNDr0Dy4+zqGFDoZtoTV4ZxNNWE=">AAACi3icdVHJSgNBEO2Me9zicvPSGARPYUYFRTyIQfAYwaiQDKGnU0kaexm6a8Q45F+86h/5N/bEHIzRgoLHq+3xKkmlcBiGn6Vgbn5hcWl5pby6tr6xWdnavncmsxya3EhjHxPmQAoNTRQo4TG1wFQi4SF5qhf1h2ewThh9h8MUYsX6WvQEZ+ipTmW3jfCCVuX1YQJWAwruRp1KNayF46CzIJqAKplEo7NV6rS7hmcKNHLJnGtFYYpxzqzfJ2FUbmcOUsafWB9aHmqmwMX5WP6IHnimS3vG+tRIx+zPiZwp54Yq8Z2K4cD9rhXkX7VWhr2zOBc6zRA0/z7UyyRFQwsvaFdY4CiHHjBuhddK+YBZxtE7NrXpLorzQlyxZup8okblA/qTKWSkqF6m+5jsG39hoP6hBZ8dSB1k3lXT9Qb6l0S/HzAL7o9q0XHt6Pakenk1ec4y2SP75JBE5JRckhvSIE3CySt5I+/kI1gPjoPz4OK7NShNZnbIVATXXzn2yvA=</latexit> Linear Threshold Unit Wiener, 1942, 1948 <latexit sha1_base64="rIu1wHcZjdVHqxpLtWaZSwQyEjo=">AAACkHicdVFNSwMxEE3X7/pV9SRegkXwVHZV0JtfFxEPCq0K7VKy2Wk3mGyWZFYsS/HXeNXf478xW3uwVgcCjzczbyZvokwKi77/WfFmZufmFxaXqssrq2vrtY3Ne6tzw6HFtdTmMWIWpEihhQIlPGYGmIokPERPl2X+4RmMFTpt4iCDULF+KnqCM3RUt7bdQXhBo4obJ8AMbSYGbKJlPOzW6n7DHwWdBsEY1Mk4brsblW4n1jxXkCKXzNp24GcYFsyg4BKG1U5uIWP8ifWh7WDKFNiwGP1hSPccE9OeNu6lSEfsz46CKWsHKnKVimFif+dK8q9cO8feSViINMsRUv49qJdLipqWhtBYGOAoBw4wboTblfKEGcbR2Tah1AzColyulJkYH6lhdY/+ZMo1MlQvk3VM9rWbkKh/aMGnGzILuXNVx85Ad5Lg9wGmwf1BIzhsHNwd1c8uxsdZJDtkl+yTgByTM3JFbkmLcPJK3sg7+fA2vRPv1Dv/LvUq454tMhHe9ReZHMzP</latexit> <latexit sha1_base64="7+HutZu99fu6873sD2pKnXxXdTc=">AAACknicdVHPTxpBFB5Waym2FrU3LpMSEw8N2UWTqietlx48YCJiAhsyOzxg4vzYzLxtIBsO/jVe7Z/jf9NZ4ABiX/ImX773c76XpFI4DMPXUrC1/WHnY/lTZffzl72v1f2De2cyy6HNjTT2IWEOpNDQRoESHlILTCUSOsnjdRHv/AHrhNF3OE0hVmykxVBwhp7qV2s9hAlalXcEaLA/aHR+2py/Z7N+tR42wrnRTRAtQZ0srdXfL/V7A8MzBRq5ZM51ozDFOGcWBZcwq/QyBynjj2wEXQ81U+DifP6LGT3yzIAOjfWukc7Z1YqcKeemKvGZiuHYvY0V5HuxbobDszgXOs0QNF8MGmaSoqGFJHQgLHCUUw8Yt8LvSvmYWcbRC7fW6S6K82K5os3a+ETNKkd0lSnWSFFN1vOYHBk/Yaz+Qwu+WZA6yLyqZuAF9CeJ3h5gE9w3G9FJo3l7Wr/8tTxOmdTId3JMIvKTXJLfpEXahJMn8kxeyN/gW3ARXAXXi9SgtKw5JGsW3PwD0MzLYw==</latexit> <latexit sha1_base64="ivYW7CFX38Z5aRf+X7iG10BBJng=">AAACgnicdVHJSgNBEO2Me9z16KUxBMRDmDGCHjyIXjwqJCokQ+jpVJImvYzdNWIY8h1e9bP8G3tiDibRgoLHq+3xKkmlcBiGX6VgaXlldW19o7y5tb2zu7d/8OhMZjk0uZHGPifMgRQamihQwnNqgalEwlMyvC3qT69gnTC6gaMUYsX6WvQEZ+ipuI3whlblTS1w3NmrhLVwEnQRRFNQIdO47+yXOu2u4ZkCjVwy51pRmGKcM4uCSxiX25mDlPEh60PLQ80UuDifqB7Tqme6tGesT410wv6eyJlybqQS36kYDtx8rSD/qrUy7F3GudBphqD5z6FeJikaWlhAu8ICRznygHErvFbKB8wyjt6omU2NKM4LccWamfOJGper9DdTyEhRvc32Mdk3/sJA/UMLvjiQOsi8q6brDfQvieYfsAgez2pRvXb2cF65vpk+Z50ckWNyQiJyQa7JHbknTcLJC3knH+QzWA5Ogyio/7QGpenMIZmJ4OobkQnHng==</latexit> McCulloch & Pitts, 1943 <latexit sha1_base64="7XYM52X1iyfLiy6SgKPzElD/dkI=">AAACmHicdVFNSysxFE3Hz1e/qu50Ey2KCykzKvjcFV2oC6GKVaEdSia9bUOTyZDceViGrv01bvW3+G9epnZhrV4IHM79ODfnRokUFn3/o+DNzM7NLyz+KS4tr6yuldY3HqxODYc611Kbp4hZkCKGOgqU8JQYYCqS8Bj1L/L84z8wVuj4HgcJhIp1Y9ERnKGjWqWdJsIzGpXd8ItUSs17tLlPawLRHtLg7OR42CqV/Yo/CjoNgjEok3HUWuuFVrOteaogRi6ZtY3ATzDMmEHBJQyLzdRCwnifdaHhYMwU2DAb/WVI9xzTph1t3IuRjtivHRlT1g5U5CoVw579nsvJn3KNFDt/w0zESYoQ80+hTiopapobQ9vCAEc5cIBxI9yulPeYYRydfROT7oMwy5fLx0zIR2pY3KNfmXyNBNXzZB2TXe0UeuoXWvDphsRC6lzVbWegO0nw/QDT4OGoEhxXjm5PytXz8XEWyTbZJQckIKekSq5IjdQJJy/klbyRd2/Lq3qX3vVnqVcY92ySifDu/gO2rM5k</latexit> Information <latexit sha1_base64="8KOpIHumxzgs/eunFRn5MRE5UPE=">AAACi3icdVHLSgNBEJys7/iKj5uXwSB4CrsqKOJBFEFvCkaFZAmzk04yZB7LTK8Yl/yLV/0j/8bZmIMx2jBQVHdPFdVJKoXDMPwsBTOzc/MLi0vl5ZXVtfXKxuaDM5nlUOdGGvuUMAdSaKijQAlPqQWmEgmPSf+y6D8+g3XC6HscpBAr1tWiIzhDT7Uq202EF7Qqv9EdY9WIHbYq1bAWjopOg2gMqmRct62NUqvZNjxToJFL5lwjClOMc2ZRcAnDcjNzkDLeZ11oeKiZAhfnI/tDuueZNvXy/mmkI/bnRs6UcwOV+ElvsOd+9wryr14jw85JnAudZgiafwt1MknR0CIL2hYWOMqBB4xb4b1S3mOWcfSJTfx0H8V5Ya74ZkI+UcPyHv3JFDZSVC+Tc0x2jVfoqX9owacXUgeZT9W0fYD+JNHvA0yDh4NadFg7uDuqnl+Mj7NIdsgu2ScROSbn5Jrckjrh5JW8kXfyEawGh8FpcPY9GpTGO1tkooKrL1D9yvs=</latexit> Information in Neuroscience Shannon, 1948 Feedback & <latexit sha1_base64="AXGNxZB2+zO5BevVdDQLAT+DUII=">AAACjXicdVHLSiNBFK20j9H4Shxw46YwCC4kdPtgIojIuNClolEhaUJ15SYprEdTdVsMbX7G7cwPzd9MdczCGL1QcDj3cW6dm6RSOAzDf6Vgbn5h8cfScnlldW19o1LdvHcmsxya3EhjHxPmQAoNTRQo4TG1wFQi4SF5uijyD89gnTD6DocpxIr1tegJztBTncpWG+EFrcpvB0xro/dpdHLUGHUqtbAejoPOgmgCamQS151qqdPuGp4p0Mglc64VhSnGObMouIRRuZ05SBl/Yn1oeaiZAhfn4w+M6K5nurRnrH8a6Zj92JEz5dxQJb5SMRy4z7mC/CrXyrDXiHOh0wxB83ehXiYpGlq4QbvCAkc59IBxK/yulA+YZRy9Z1OT7qI4L5YrxkzJJ2pU3qUfmWKNFNXLdB2TfeMVBuobWvDZhtRB5l01XW+gP0n0+QCz4P6gHh3WD26Oaue/J8dZIttkh+yRiPwi5+SKXJMm4eSVvJE/5G+wERwHp8HZe2lQmvT8JFMRXP4HgH7KgA==</latexit> Homeostasis <latexit sha1_base64="Ob8a8z66PKAuvMI8cWjpQuYNPbY=">AAACjnicdVHLSgMxFE3Hd31VXbhwEyyCqzKjom5E0Y3uFKwK7VAy6W0bzGNI7ohlmK9xqx/k35ipXVirFwKHc18n5yapFA7D8LMSzMzOzS8sLlWXV1bX1msbmw/OZJZDkxtp7FPCHEihoYkCJTylFphKJDwmz1dl/vEFrBNG3+MwhVixvhY9wRl6qlPbbiO8olX5je4Zq0YsFbro1OphIxwFnQbRGNTJOG47G5VOu2t4pkAjl8y5VhSmGOfMouASimo7c5Ay/sz60PJQMwUuzkc/KOieZ7rUK/BPIx2xPztyppwbqsRXeo0D9ztXkn/lWhn2TuNc6DRD0Px7US+TFA0t7aBdYYGjHHrAuBVeK+UDZhlHb9rEpPsozktx5ZiJ9Ykqqnv0J1PKSFG9TtYx2Td+w0D9Qws+3ZA6yLyrpusN9CeJfh9gGjwcNKLDxsHdUf3icnycRbJDdsk+icgJuSDX5JY0CScFeSPv5COoBcfBWXD+XRpUxj1bZCKC6y/bR8wQ</latexit> <latexit sha1_base64="Wb6y7ZSTddS6w2I60ldb4wD0qSE=">AAACi3icdVHLSgMxFE3Hd33Vx85NsCiuyowKirgoiuJSwarQDiWTuW2DyWRI7kjr0H9xq3/k35ipXVirFy4czn0dzo1SKSz6/mfJm5mdm19YXCovr6yurVc2Nh+szgyHBtdSm6eIWZAigQYKlPCUGmAqkvAYPV8W9ccXMFbo5B4HKYSKdRPREZyho9qV7RZCH43KrwHiiPFn2toftitVv+aPgk6DYAyqZBy37Y1SuxVrnilIkEtmbTPwUwxzZlBwCcNyK7OQuu2sC00HE6bAhvlI/pDuOSamHW1cJkhH7M+JnClrBypynYphz/6uFeRftWaGndMwF0maIST8+1AnkxQ1LbygsTDAUQ4cYNwIp5XyHjOMo3NsYtN9EOaFuGLNxPlIDct79CdTyEhR9Sf7mOxqd6Gn/qEFnx5ILWTOVR07A91Lgt8PmAYPh7XgqHZ4d1ytX4yfs0h2yC45IAE5IXVyQ25Jg3DySt7IO/nwVr0j78w7/271SuOZLTIR3tUXgDrKHA==</latexit> <latexit sha1_base64="7Z19sU/anYqKitl902rCvG9IDeI=">AAACi3icdVFdSwJBFB23L7Mvrd56GZKgJ9mtoIgepAh8LMgKdJHZ8apDMzvLzN3IFv9Lr/WP+jfNqg+adeHC4dyvw7lRIoVF3/8ueEvLK6trxfXSxubW9k65svtodWo4NLmW2jxHzIIUMTRRoITnxABTkYSn6OUmrz+9grFCxw84TCBUrB+LnuAMHdUp77cR3tCorKEVaIvMCjvqlKt+zR8HXQTBFFTJNO46lUKn3dU8VRAjl8zaVuAnGGbMoOASRqV2aiFh/IX1oeVgzBTYMBvLH9Ejx3RpTxuXMdIxOzuRMWXtUEWuUzEc2N+1nPyr1kqxdxFmIk5ShJhPDvVSSVHT3AvaFQY4yqEDjBvhtFI+YIZxdI7NbXoIwiwXl6+ZOx+pUemIzjK5jATV23wfk33tLgzUP7TgiwOJhdS5qrvOQPeS4PcDFsHjSS04rZ3cn1Xr19PnFMkBOSTHJCDnpE4a5I40CSfv5IN8ki9vyzv1Lr2rSatXmM7skbnwbn8AY8bLBA==</latexit> <latexit sha1_base64="EQCN1geucAdf2sTINZWHMkTu1kk=">AAACjHicdVHLSgMxFE3HV62vqrhyEyyCqzKjgoIIRUFcSQWrQjuUTHrbBpPJkNwRy9CPcatf5N+YabvoQy8EDue+Ts6NEiks+v5PwVtaXlldK66XNja3tnfKu3vPVqeGQ4Nrqc1rxCxIEUMDBUp4TQwwFUl4id5u8/zLOxgrdPyEgwRCxXqx6ArO0FHt8kEL4QONyh4gNdpyATGHYbtc8av+KOgiCCagQiZRb+8W2q2O5qmCGLlk1jYDP8EwYwYFlzAstVILCeNvrAdNB2OmwIbZSP+QHjumQ7vauBcjHbHTHRlT1g5U5CoVw76dz+XkX7lmit3LMBNxkqL713hRN5UUNc3NoB1hgKMcOMC4EU4r5X1mGEdn2cykpyDMcnH5mJn1kRqWjuk0k8tIUH3M1jHZ025DX/1DC77YkFhInau64wx0JwnmD7AInk+rwVn19PG8UruZHKdIDskROSEBuSA1ck/qpEE4ycgn+SLf3rZ37l151+NSrzDp2Scz4d39Aj/7y2I=</latexit> Ashby, 1952 Barlow, 1961, 1989 Bialek et al., 1991 <latexit sha1_base64="0wziz9IYdUYmAIW6a8R0uXe72Dg=">AAACi3icdVHJSgNBEO2Me9zicvPSGAQPEmai4oIHFwSPCkaFZAg9nUrSpHt66K4R45B/8ap/5N/YE3MwRgsKHq+2x6sokcKi738WvKnpmdm5+YXi4tLyymppbf3B6tRwqHEttXmKmAUpYqihQAlPiQGmIgmPUe8qrz8+g7FCx/fYTyBUrBOLtuAMHdUsbTYQXtCo7MJ2o/4eDU4Oq4NmqexX/GHQSRCMQJmM4ra5Vmg2WpqnCmLkkllbD/wEw4wZFFzCoNhILSSM91gH6g7GTIENs6H8Ad1xTIu2tXEZIx2yPycypqztq8h1KoZd+7uWk3/V6im2j8NMxEmKEPPvQ+1UUtQ094K2hAGOsu8A40Y4rZR3mWEcnWNjm+6DMMvF5WvGzkdqUNyhP5lcRoLqZbyPyY52F7rqH1rwyYHEQupc1S1noHtJ8PsBk+ChWgn2K9W7g/L55eg582SLbJNdEpAjck5uyC2pEU5eyRt5Jx/esrfvnXpn361eYTSzQcbCu/4CTWrJiQ==</latexit> <latexit sha1_base64="sjWZM1qHSLACHr+0fY1cKrKKRIg=">AAACknicdVG5TgMxEHWWK4QrHB2NRYREgaJdQFxVgIaCAiQCSMkq8jqTxMJer+xZIFql4Gto4XP4G7whBSEw0lhPb06/iRIpLPr+Z8Gbmp6ZnSvOlxYWl5ZXyqtrd1anhkOda6nNQ8QsSBFDHQVKeEgMMBVJuI8eL/L4/RMYK3R8i/0EQsW6segIztBRrfJmE+EFjcrOmZH6eZcGJ4dB/h6fDFrlil/1h0YnQTACFTKy69ZqodVsa54qiJFLZm0j8BMMM2ZQcAmDUjO1kDD+yLrQcDBmCmyYDX8xoNuOadOONs5jpEP2Z0XGlLV9FblMxbBnf8dy8q9YI8XOcZiJOEkRYv49qJNKiprmktC2MMBR9h1g3Ai3K+U9ZhhHJ9xYp9sgzPLl8jZj4yM1KG3Tn0y+RoLqZTyPya52E3rqH1rwyYLEQupU1W0noDtJ8PsAk+BurxrsV/duDiq189FximSTbJEdEpAjUiOX5JrUCSev5I28kw9vwzv1zryL71SvMKpZJ2PmXX0B1pHLZg==</latexit> Feedback <latexit sha1_base64="sDu0ncCbDM9WG12Q7DXnhJXwTqM=">AAACk3icdVFNSyNBEO2M7qrZr+iyp/XQGIQ9LGEmK6h4kbgHLwsKRoVkCDWdStKke3rorhHDkIu/xuv6b/w39sQcjNGChserj1f9KsmUdBSGj5VgZfXDx7X1jeqnz1++fqttbl06k1uBbWGUsdcJOFQyxTZJUnidWQSdKLxKxidl/uoGrZMmvaBJhrGGYSoHUgB5qlfb7hLektVFS4LCMUfioBq/eXR4GE17tXrYCGfBl0E0B3U2j7PeZqXX7RuRa0xJKHCuE4UZxQVYkkLhtNrNHWYgxjDEjocpaHRxMfvGlO96ps8HxvqXEp+xLzsK0M5NdOIrNdDIvc6V5Fu5Tk6Dg7iQaZYTpuJZaJArToaXnvC+tChITTwAYaXflYsRWBDknVuYdBHFRblcOWZBPtHT6i5/yZRrZKRvF+tADY1XGOl3aCmWGzKHuXfV9L2B/iTR6wMsg8tmI/rTaJ7v1Y9b8+Oss59sh/1iEdtnx+yUnbE2E+yO3bP/7CH4ERwFreDvc2lQmfd8ZwsR/HsCu3TMQQ==</latexit> <latexit sha1_base64="Gm6/O48VljMTj7pi0hOM4GOGzkE=">AAACiHicdVHLSgMxFE3Hd321unQTLIKrMqNCdScK4lLBPqAdSiZz2waTyZDcUcvQT3Gr3+TfmKld2FYvXDic+zqcG6VSWPT9r5K3srq2vrG5Vd7e2d3br1QPWlZnhkOTa6lNJ2IWpEigiQIldFIDTEUS2tHzbVFvv4CxQidPOE4hVGyYiIHgDB3Vr1R7CG9oVH4HEEeMP0/6lZpf96dBl0EwAzUyi4d+tdTvxZpnChLkklnbDfwUw5wZFFzCpNzLLKRuNRtC18GEKbBhPtU+oSeOielAG5cJ0in7eyJnytqxilynYjiyi7WC/KvWzXBwGeYiSTOEhP8cGmSSoqaFETQWBjjKsQOMG+G0Uj5ihnF0ds1tegrCvBBXrJk7H6lJ+YT+ZgoZKaq3+T4mh9pdGKl/aMGXB1ILmXNVx85A95Jg8QHLoHVWD87rZ48Xteub2XM2yRE5JqckIA1yTe7JA2kSTl7JO/kgn17Z872Gd/XT6pVmM4dkLrybb6rhyVw=</latexit> Perception & Control Perceptron <latexit sha1_base64="dS0o4qoTLrOqaJnrW4RejCy0TaQ=">AAAClXicdVFNSyNBEO2Mump01+gePAjSGJQ9hRkV9CLIuog3IxgVkiH0dCpJY3/RXSOGITd/zV53/8z+m+2JORijBQ2PV1Wvql9lVgqPcfyvEi0sLn1ZXlmtrq1//bZR29y68yZ3HFrcSOMeMuZBCg0tFCjhwTpgKpNwnz1elPn7J3BeGH2LIwupYgMt+oIzDFS3tttBeEaniiYEQVuStHNAL4xGZ+S4W6vHjXgSdB4kU1An02h2NyvdTs/wXIFGLpn37SS2mBbMoeASxtVO7sEy/sgG0A5QMwU+LSYfGdP9wPRo37jwNNIJ+7ajYMr7kcpCpWI49O9zJflRrp1j/zQthLY5guavg/q5pGho6QrtCQcc5SgAxp0Iu1I+ZI5xDN7NKN0maVEuV8rMjM/UuLpP3zLlGhbV82wdkwMTJgzVJ7Tg8w3WQx5cNb1gYDhJ8v4A8+DusJEcNQ5vjuvnP6fHWSE7ZI/8IAk5IefkijRJi3DyQn6TP+RvtB2dRb+iy9fSqDLt+U5mIrr+D85nzpU=</latexit> MacKay, 1956 Kalman, 1960 <latexit sha1_base64="ZwzkvT+u653VSsj6A9nXBi4ioiQ=">AAACinicdVHLSitBEO2M3qtGvUZdumkMwl2FmSiouBF14TKCUSEZQk+nkjT2i+4aMQz5Frf6Sf6NPTELY7Sg4HDqdTiVWSk8xvF7JVpa/vN3ZXWtur6x+W+rtr1z503uOLS5kcY9ZMyDFBraKFDCg3XAVCbhPnu8LOv3T+C8MPoWxxZSxYZaDARnGKhebbeL8IxOFS0ICy06oye9Wj1uxNOgiyCZgTqZRau3Xel1+4bnCjRyybzvJLHFtGAOBZcwqXZzD5bxRzaEToCaKfBpMVU/oQeB6dOBcSE10in7daJgyvuxykKnYjjy32sl+VOtk+PgJC2EtjmC5p+HBrmkaGhpBe0LBxzlOADGnQhaKR8xxzgGw+Y23SZpUYor18ydz9SkekC/MqUMi+p5vo/JoQkXRuoXWvDFAeshD66afjAwvCT5/oBFcNdsJIeN5s1R/fxi9pxVskf2yX+SkGNyTq5Ji7QJJ2PyQl7JW7QRNaPT6OyzNarMZnbJXERXH1RDyo0=</latexit> <latexit sha1_base64="MGXQrLMNzFUiERn3W0kO7UB0sd8=">AAACjHicdVFdSxtBFJ2s1mqsNlb65MtgEHwoYddatYggFUQQQcGokCzh7uQmGZzZWWbuimHJj/HV/qL+G2djHoypFy4czv06nJtkSjoKw3+VYG7+08LnxaXq8peV1a+1tW83zuRWYFMYZexdAg6VTLFJkhTeZRZBJwpvk/uTsn77gNZJk17TMMNYQz+VPSmAPNWpfW8TPpLVxQWIcxj+4NHvX3ujTq0eNsJx8FkQTUCdTeKys1bptLtG5BpTEgqca0VhRnEBlqRQOKq2c4cZiHvoY8vDFDS6uBjrH/Etz3R5z1ifKfEx+3aiAO3cUCe+UwMN3PtaSf6v1sqpdxAXMs1ywlS8HurlipPhpRm8Ky0KUkMPQFjptXIxAAuCvGVTm66juCjFlWumzid6VN3ib5lSRkb6cboPVN/4CwP9AS3F7EDmMPeumq430L8kev+AWXCz04h+NnauduvHfybPWWQbbJNts4jts2N2xi5ZkwlWsCf2zP4Gq8FucBgcvbYGlcnMOpuK4PQFBLTJ1g==</latexit> Rosenblatt, 1958 <latexit sha1_base64="ybTh9qYo2ymllaH2IIJ1tNBYj0I=">AAACjHicdVFbSxtBFJ6srZfUS1LxqS9Dg+BDCbsqtqUI0kIp9MWCUSFZwtnJSTI4l2XmbElY8mP6Wn9R/01nYx6M0QMHPr5z+/hOlivpKY7/1aK1V6/XNza36m+2d3b3Gs23194WTmBHWGXdbQYelTTYIUkKb3OHoDOFN9ndt6p+8xudl9Zc0TTHVMPIyKEUQIHqNw56hBNyuvwJSoP5wJPPZ/Gs32jF7XgefBUkC9Bii7jsN2v93sCKQqMhocD7bhLnlJbgSAqFs3qv8JiDuIMRdgM0oNGn5Vz/jB8GZsCH1oU0xOfs44kStPdTnYVODTT2T2sV+VytW9DwU1pKkxeERjwcGhaKk+WVGXwgHQpS0wBAOBm0cjEGB4KCZUubrpK0rMRVa5bOZ3pWP+SPmUpGTnqy3AdqZMOFsX6BlmJ1IPdYBFftIBgYXpI8fcAquD5uJyft41+nrYuvi+dssnfsPTtiCfvILtgPdsk6TLCS/WF/2X20G51GX6Lzh9aotpjZZ0sRff8POZnJ7w==</latexit> <latexit sha1_base64="Qy9ToeQXBXxtgV/63aK2tg1yA/0=">AAACkHicdVFNbxoxEDXbJCU0TQk9RblYRZFyiNAuaVV6Km0uUU6kCiQSrJDXDGDFHyt7tgKtUH9Nr+3v6b+pFzgEaEey9PTm443fJKkUDsPwTyl4sbd/8LJ8WHl19Pr4TfWk1nMmsxy63EhjHxPmQAoNXRQo4TG1wFQi4SF5ui7yD9/BOmH0Pc5TiBWbaDEWnKGnhtXTAcIMrcq/GQc6kQzxkkafPrQWw2o9bITLoLsgWoM6WUdneFIaDkaGZwo0csmc60dhinHOLAouYVEZZA5Sxp/YBPoeaqbAxfnyDwt67pkRHRvrn0a6ZJ935Ew5N1eJr1QMp247V5D/yvUzHLfiXOg0Q9B8JTTOJEVDC0PoSFjgKOceMG6F35XyKbOMo7dtY9J9FOfFcsWYDflELSrn9DlTrJGimm3WMTkxXmGq/kMLvtuQOsi8q2bkDfQnibYPsAt6zUZ01Wjeva+3v66PUyZn5B25IBH5SNrkhnRIl3Dyg/wkv8jvoBa0gs/Bl1VpUFr3vCUbEdz+Bbp6y+g=</latexit> Perceptual Control Powers, 1973 <latexit sha1_base64="HjLxr0bZvBZLN10r2jNpmQEsUw0=">AAACinicdVHLSgMxFE3Hd31Vu3QTLIKrMlMFFTeiLlxWsCq0Q8mkt20wmYTkjliGfotb/ST/xkzbhbV64cLh3Nfh3MRI4TAMv0rB0vLK6tr6Rnlza3tnt7K3/+h0Zjm0uJbaPifMgRQptFCghGdjgalEwlPyclPUn17BOqHTBxwZiBUbpKIvOENPdSvVDsIbWpU3wS80mDE57lZqYT2cBF0E0QzUyCya3b1St9PTPFOQIpfMuXYUGoxzZlFwCeNyJ3NgGH9hA2h7mDIFLs4n6sf0yDM92tfWZ4p0wv6cyJlybqQS36kYDt3vWkH+VWtn2D+Pc5GaDCHl00P9TFLUtLCC9oQFjnLkAeNWeK2UD5llHL1hc5seojgvxBVr5s4nalw+oj+ZQoZB9Tbfx+RA+wtD9Q8t+OKAcZB5V3XPG+hfEv1+wCJ4bNSjk3rj/rR2dT17zjo5IIfkmETkjFyRO9IkLcLJiLyTD/IZbAWN4CK4nLYGpdlMlcxFcPsNOUTKgA==</latexit> <latexit sha1_base64="5k3xzP/fYxkjDiIboJ6NPBbROYc=">AAACh3icdVHJTgJBEG3GHTfUo5eOxMQTzqhRjy4Xj5oImMCE9DQFdOxl0l1DJBP+xKv+k39jD3AQ0UoqeXm1vbxKUikchuFXKVhaXlldW98ob25t7+xW9vYbzmSWQ50baexLwhxIoaGOAiW8pBaYSiQ0k9f7ot4cgnXC6GccpRAr1teiJzhDT3UqlTbCG1qV3xuN1shxp1INa+Ek6CKIZqBKZvHY2St12l3DMwUauWTOtaIwxThnFgWXMC63Mwcp46+sDy0PNVPg4nwifUyPPdOlPWN9aqQT9udEzpRzI5X4TsVw4H7XCvKvWivD3nWcC51mCJpPD/UySdHQwgfaFRY4ypEHjFvhtVI+YJZx9G7NbXqO4rwQV6yZO5+ocfmY/mQKGSmqt/k+JvvGXxiof2jBFwdSB5l31XS9gf4l0e8HLILGWS06r509XVRv7mbPWSeH5IickIhckRvyQB5JnXAyJO/kg3wGG8FpcBlcT1uD0mzmgMxFcPsNNRPJLg==</latexit> Probabilistic Models / Backprop & Statistical Mechanics Werbos, 1974 Rumelhart et al., 1986 Ackley et al., 1985 Pearl, 1986 <latexit sha1_base64="aN6wHxHdd7V/+o09bGbWc3bGTyE=">AAAClnicdVHLSgMxFE3Hd31V3QguDBbBVZ1RQVciiuhGqGBVaIeSSW/bYDIZkjtiGWbp17jVj/FvzNQurNULgcO5j3NzbpRIYdH3P0ve1PTM7Nz8QnlxaXlltbK2fm91ajg0uJbaPEbMghQxNFCghMfEAFORhIfo6aLIPzyDsULHdzhIIFSsF4uu4Awd1a5stxBe0KisbnTEIlFICk5vdAekpft5u1L1a/4w6CQIRqBKRlFvr5XarY7mqYIYuWTWNgM/wTBjxs2VkJdbqYWE8SfWg6aDMVNgw2z4k5zuOqZDu9q4FyMdsj87MqasHajIVSqGffs7V5B/5Zopdk/CTMRJihDzb6FuKilqWthCO8IARzlwgHEjCg94nxnG0Zk3NukuCLNiuWLMmHyk8vIu/ckUaySoXsbrmOxpp9BX/9CCTzYkFlLnqrtKXnYnCX4fYBLcH9SCw9rB7VH17Hx0nHmyRXbIHgnIMTkj16ROGoSTV/JG3smHt+mdepfe1XepVxr1bJCx8Opfxz/O/Q==</latexit> <latexit sha1_base64="u+mGdzI2I6m/Z55/mMsj6gfhEuo=">AAACi3icdVFNSwMxEE3X7/pVP25egkXxVHZVUMSDKIJHBatCu5RsOm1Dk01IZsW69L941X/kvzFbe7BWBwKPNzN5jzeJkcJhGH6WgpnZufmFxaXy8srq2nplY/PB6cxyqHMttX1KmAMpUqijQAlPxgJTiYTHpH9V9B+fwTqh03scGIgV66aiIzhDT7Uq202EF7Qqv2S8b6w2tLk/bFWqYS0cFZ0G0RhUybhuWxulVrOteaYgRS6Zc40oNBjnzKLgEoblZubAeAXWhYaHKVPg4nxkf0j3PNOmHW39S5GO2J8bOVPODVTiJxXDnvvdK8i/eo0MO6dxLlKTIaT8W6iTSYqaFlnQtrDAUQ48YNwK75XyHrOMo09s4qf7KM4Lc8U3E/KJGpb36E+msGFQvUzOMdnVXqGn/qEFn14wDjKfqm77AP1Jot8HmAYPh7XoqHZ4d1y9uBwfZ5HskF1yQCJyQi7IDbkldcLJK3kj7+QjWA2OgrPg/Hs0KI13tshEBddf3k7KSQ==</latexit> <latexit sha1_base64="nxTdOmVhGeA9GOp4kQv37Cl+R+8=">AAACjHicdVFNSwMxEE3X7/rVKp68BIvgQcquCioiiIJ4rGBVaJeSTadtaLJZklltWfpjvOov8t+YrT1YqwOBx5s3mcebKJHCou9/Fry5+YXFpeWV4ura+sZmqbz1aHVqONS5lto8R8yCFDHUUaCE58QAU5GEp6h/k/efXsBYoeMHHCYQKtaNRUdwho5qlXaaCAM0KqvpVyc7pMH56fGoVar4VX9cdBYEE1Ahk6q1yoVWs615qiBGLpm1jcBPMMyYQcEljIrN1ELCeJ91oeFgzBTYMBv7H9F9x7RpRxv3YqRj9udExpS1QxU5pWLYs797OflXr5Fi5yzMRJykCDH/XtRJJUVN8zBoWxjgKIcOMG6E80p5jxnG0WUx9dNDEGa5ufybqfWRGhX36U8mt5GgGkzrmOxqt6Gn/qEFnx1ILKQuVd12AbqTBL8PMAsej6rBcfXo/qRydT05zjLZJXvkgATklFyRO1IjdcJJRt7IO/nwNrwT78K7/JZ6hcnMNpkq7/YLnqXKHw==</latexit> Spatiotemporal Predictive Coding Srinivasan et al., 1982 Dong & Atick, 1995 <latexit sha1_base64="LHKDIoF42lxXODC1f2Tbrf6Ne+w=">AAAClXicdVHLSitBEO2M79yrRl24EKS5QXAVZlTQjSAq4kZUNCokQ+jpVJLGfgzdNWIYsvNr3OrP+Df2xCyM8RY0HM6pV59KUikchuFHKZianpmdm18o//m7uLRcWVm9cyazHOrcSGMfEuZACg11FCjhIbXAVCLhPnk8KfT7J7BOGH2L/RRixbpadARn6KlWZbOJ8IxW5TfoGYdekPQCeI9pwd2gVamGtXAYdBJEI1Alo7hqrZRazbbhmQKNXDLnGlGYYpwz61tLGJSbmYOU8UfWhYaHmilwcT78yIBueaZNO8b6p5EO2e8VOVPO9VXiMxXDnvupFeRvWiPDzkGcC51mCJp/DepkkqKhhSu0LSxwlH0PGLfC70q9BZZx9N6NdbqN4rxYrmgzNj5Rg/IW/c4Ua6SonsfzmOwaP6Gn/kMLPlmQOsi8q6btDfQniX4eYBLc7dSi3drO9V716Hh0nHmyQf6RbRKRfXJEzskVqRNOXsgreSPvwXpwGJwGZ1+pQWlUs0bGIrj8BMPYzwk=</latexit> <latexit sha1_base64="HhJ5aAWGTaSElLvgPqXPcBaQt0A=">AAACjHicdVFdSwJBFB23L7MvK3rqZUiCHkJ2S6iIQAqix4LMQBeZHa86OLOzzNyNZPHH9Fq/qH/TrPmQWRcuHM79OpwbJVJY9P3PgrewuLS8Ulwtra1vbG6Vt3eerE4NhwbXUpvniFmQIoYGCpTwnBhgKpLQjIY3eb35AsYKHT/iKIFQsX4seoIzdFSnvNdGeEWjsiaYSNtjGlyc1cadcsWv+pOg8yCYggqZxn1nu9BpdzVPFcTIJbO2FfgJhhkzKLiEcamdWkgYH7I+tByMmQIbZhP9Y3romC7taeMyRjphf05kTFk7UpHrVAwH9nctJ/+qtVLsnYeZiJMUIebfh3qppKhpbgbtCgMc5cgBxo1wWikfMMM4OstmNj0GYZaLy9fMnI/UuHRIfzK5jATV62wfk33tLgzUP7Tg8wOJhdS5qrvOQPeS4PcD5sHTSTU4rZ481Cr16+lzimSfHJAjEpAzUid35J40CCcZeSPv5MPb9GrepXf13eoVpjO7ZCa82y+DPcoS</latexit> <latexit sha1_base64="RnXwjKHLRK0dHWSs3D8EJuMK6e4=">AAACjnicdVHLSgMxFE3Hd31VXbhwEyyCqzKjom5E0Y1LRatCO5RMetsGk0lI7ohlmK9xqx/k35ipXVirFwKHcx/n5tzESOEwDD8rwczs3PzC4lJ1eWV1bb22sfngdGY5NLmW2j4lzIEUKTRRoIQnY4GpRMJj8nxV5h9fwDqh03scGogV66eiJzhDT3Vq222EV7QqvzMlg6CMtkwWnVo9bISjoNMgGoM6GcdNZ6PSaXc1zxSkyCVzrhWFBuOcWRRcQlFtZw4M48+sDy0PU6bAxfnoBwXd80yX9rT1L0U6Yn925Ew5N1SJr1QMB+53riT/yrUy7J3GuUhNhpDyb6FeJilqWtpBu8ICRzn0gHEr/K6UD5hlHL1pE5PuozgvlyvHTMgnqqju0Z9MuYZB9TpZx2Rfe4WB+ocWfLrBOMi8q7rrDfQniX4fYBo8HDSiw8bB7VH94nJ8nEWyQ3bJPonICbkg1+SGNAknBXkj7+QjqAXHwVlw/l0aVMY9W2QigusvkQHMZw==</latexit> <latexit sha1_base64="vz/wr/GI3GzpLDcFxw8a27I8Iew=">AAACk3icdVHLSgMxFE3Hd31VxZUugkVwIWXGBypufC3cCApWhXYomfS2DU0mQ3JHLEM3fo1b/Rv/xkztwlq9EDic+zg350aJFBZ9/7PgTUxOTc/MzhXnFxaXlksrqw9Wp4ZDlWupzVPELEgRQxUFSnhKDDAVSXiMupd5/vEZjBU6vsdeAqFi7Vi0BGfoqEZps47wgkZl57wroUcBKZOVXRqcHB/2G6WyX/EHQcdBMARlMozbxkqhUW9qniqIkUtmbS3wEwwzZlBwCf1iPbWQMN5lbag5GDMFNswG3+jTbcc0aUsb92KkA/ZnR8aUtT0VuUrFsGN/53Lyr1wtxdZxmIk4SRFi/i3USiVFTXNPaFMY4Ch7DjBuhNuV8g4zjKNzbmTSfRBm+XL5mBH5SPWL2/Qnk6+RoHoZrWOyrZ1CR/1DCz7ekFhInau66Qx0Jwl+H2AcPOxVgv3K3t1B+exieJxZskG2yA4JyBE5I9fkllQJJ6/kjbyTD2/dO/UuvKvvUq8w7FkjI+HdfAHlzcxV</latexit> <latexit sha1_base64="o2HoSuSdBq83j37siXbvMzYKLCg=">AAAClnicdVFNSwMxEE3X7/pV9SJ4MFgED1J2Vfy4iCiixypWhXYp2XTaBpPNksyKZenRX+NVf4z/xmztwVodCDzezLyZvIkSKSz6/mfBm5icmp6ZnSvOLywuLZdWVu+tTg2HGtdSm8eIWZAihhoKlPCYGGAqkvAQPV3k+YdnMFbo+A57CYSKdWLRFpyho5qlzQbCCxqV3aYKZJcZpICUycouDU6OD/vNUtmv+IOg4yAYgjIZRrW5Umg2Wpo7tRi5ZNbWAz/BMHPKgkvoFxuphYTxJ9aBuoMxU2DDbPCTPt12TIu2tXEvRjpgf3ZkTFnbU5GrVAy79ncuJ//K1VNsH4eZiJMUIebfg9qppKhpbgttCQMcZc8Bxo1wu1Lu3GAcnXkjSndBmOXL5TIj4yPVL27Tn0y+RoLqZbSOyY52E7rqH1rw8YbEQupc1S1noDtJ8PsA4+B+rxLsV/ZuDspn58PjzJINskV2SECOyBm5JlVSI5y8kjfyTj68de/Uu/Suvku9wrBnjYyEV/0CTl/Nzw==</latexit> <latexit sha1_base64="j90RZquCLRJM3MdwD7TydOvl1Oc=">AAACkXicdVHLSiNBFK30jBrjKxl3zqYwCK5CtzOg7oJuhNlkwCRC0oTq6puksB5N1W0xNAG/xu3M7/g3U53phTEzFwoO5z7OrXOTTAqHYfhWCz593treqe829vYPDo+arS8DZ3LLoc+NNPYhYQ6k0NBHgRIeMgtMJRKGyeNtmR8+gXXC6HtcZBArNtNiKjhDT02aJ2OEZ7Sq6FlIBUfxBPTWpELPlpNmO+yEq6CbIKpAm1TRm7Rqk3FqeK5AI5fMuVEUZhgXzKLgEpaNce4gY/yRzWDkoWYKXFysPrGkZ55J6dRY/zTSFfu+o2DKuYVKfKViOHcfcyX5r9wox+lVXAid5Qia/xWa5pKioaUjNBUWOMqFB4xb4XelfM4s4+h9W5t0H8VFuVw5Zk0+UcvGGX3PlGtkqJ7X65icGa8wV/+hBd9syBzk3lWTegP9SaKPB9gEg4tO9K1z8fN7u3tTHadOvpJTck4ickm65I70SJ9w8kJeyS/yOzgOroNuUNUGtarnmKxF8OMPhtLNNA==</latexit> <latexit sha1_base64="k6aDsxtZ1TnGnzqnmxmix3ioiJ0=">AAACi3icdVHLSgMxFE3Hd33Vx85NsAgupMyoaBUXogguK1gV2qFk0ts2mEyG5I60Dv0Xt/pH/o2Z2oW1euHC4dzX4dwokcKi738WvJnZufmFxaXi8srq2nppY/PB6tRwqHMttXmKmAUpYqijQAlPiQGmIgmP0fN1Xn98AWOFju9xkECoWDcWHcEZOqpV2m4i9NGorAbMyAManFVPhq1S2a/4o6DTIBiDMhlHrbVRaDXbmqcKYuSSWdsI/ATDjBkUXMKw2EwtJIw/sy40HIyZAhtmI/lDuueYNu1o4zJGOmJ/TmRMWTtQketUDHv2dy0n/6o1UuxUw0zESYoQ8+9DnVRS1DT3graFAY5y4ADjRjitlPeYYRydYxOb7oMwy8XlaybOR2pY3KM/mVxGgqo/2cdkV7sLPfUPLfj0QGIhda7qtjPQvST4/YBp8HBYCY4qh3fH5cur8XMWyQ7ZJfskIKfkktySGqkTTl7JG3knH96qd+SdexffrV5hPLNFJsK7+QJVu8mN</latexit> Variational Inference Reinforcement Hinton & Van Camp, 1993 Neal & Hinton, 1998 Sutton & Barto, 1998 <latexit sha1_base64="9oRcd11qYqL6kjoEYYM1Va/fcPA=">AAACl3icdVFdSxtBFJ1sW7Wx2qR9Kn0ZGgQfJOzGgvqktFB8VGpUSJZwd3KTDJmPZeauJCx57a/pa/tf+m86G/NgTHth4HDux7lzbpYr6SmO/9SiFy9fbW3vvK7vvtnbf9tovrv1tnACu8Iq6+4z8KikwS5JUnifOwSdKbzLpl+r/N0DOi+tuaF5jqmGsZEjKYACNWjwPuGMnC6/O2nkA3gwHImDah/x5Oy0sxg0WnE7XgbfBMkKtNgqrgbN2qA/tKLQaEgo8L6XxDmlJTiSQuGi3i885iCmMMZegAY0+rRcfmXBDwIz5CPrwjPEl+zTjhK093OdhUoNNPHPcxX5r1yvoNFpWkqTF4RGPAqNCsXJ8soXPpQOBal5ACCcDLtyMQEHgoJ7a5NukrSslqvGrMlnelE/4E+Zao2c9Gy9DtTYBoWJ/g8txWZD7rEIrtphMDCcJHl+gE1w22knx+3O9efWxZfVcXbYR/aJHbKEnbALdsmuWJcJ9oP9ZL/Y7+hDdB59iy4fS6Paquc9W4vo+i9Ugc4/</latexit> <latexit sha1_base64="Rmp7t8UJ9vZQM5Yl0fY1Z15t+ys=">AAACi3icdVHLSitBEO2Mj6u5PuJj56YxCHcVZlTwcnEhiuBSwUQhGUJNp5I0dk8P3TUX45B/cat/5N/YE7MwRgsKDqdeh1NJpqSjMHyrBAuLS8u/Vlarv9fWNzZrW9stZ3IrsCmMMvY+AYdKptgkSQrvM4ugE4V3ycNFWb/7j9ZJk97SKMNYwyCVfSmAPNWt7XYIH8nqogVWTjhQ426tHjbCSfB5EE1BnU3jurtV6XZ6RuQaUxIKnGtHYUZxAZakUDiudnKHGYgHGGDbwxQ0uriYyB/zA8/0eN9YnynxCft5ogDt3EgnvlMDDd3XWkl+V2vn1P8bFzLNcsJUfBzq54qT4aUXvCctClIjD0BY6bVyMQQLgrxjM5tuo7goxZVrZs4nelw94J+ZUkZG+nG2D9TA+AtD/QMtxfxA5jD3rpqeN9C/JPr6gHnQOmxER43Dm+P62fn0OStsj+2zPyxiJ+yMXbFr1mSCPbFn9sJeg/XgKPgXnH60BpXpzA6bieDyHTgmyu8=</latexit> <latexit sha1_base64="m5zXe0czCWe6YtAevEIPu/0M2Es=">AAACk3icdVFNTxsxEHW25aOBQijqiR6sRqAeULQbQDTiQikHLkhUIoCUrCKvM0ms+GNlz1aJVrnwa7jCv+m/qTfkQAiMZOnpzcyb8ZsklcJhGP4rBR8+Li2vrH4qr61/3tisbH25cSazHJrcSGPvEuZACg1NFCjhLrXAVCLhNhn+LvK3f8E6YfQ1jlOIFetr0ROcoac6lW9thBFalZ8b3aftPfoLBR/u06jROJp0KtWwFk6DLoJoBqpkFledrVKn3TU8U6CRS+ZcKwpTjHNmvaqESbmdOUgZH7I+tDzUTIGL8+k3JnTXM13aM9Y/jXTKvuzImXJurBJfqRgO3OtcQb6Va2XY+xnnQqcZgubPg3qZpGho4QntCgsc5dgDxq3wu1I+YJZx9M7NKV1HcV4sV8jMjU/UpLxLXzLFGimq0Xwdk33jJwzUO7Tgiw2pg8y7arreQH+S6PUBFsFNvRYd1Op/DqunZ7PjrJId8p38IBE5JqfkglyRJuHknjyQR/IUfA1OgrPg/Lk0KM16tslcBJf/AXEqzB8=</latexit> <latexit sha1_base64="1fsGlf9WV8fdkC9EcdHzNT+oS1M=">AAACiXicdVHLSgMxFE3Hd31VXboJFsFVmVFBcSW60V0Fq0I7lEx6pw3mMSR3pGXor7jVX/JvzLRdWKsXAodzXyfnJpkUDsPwqxIsLa+srq1vVDe3tnd2a3v7T87klkOLG2nsS8IcSKGhhQIlvGQWmEokPCevt2X++Q2sE0Y/4iiDWLG+FqngDD3Vre13EIZoVXGvU7CgOYy7tXrYCCdBF0E0A3Uyi2Z3r9Lt9AzPFWjkkjnXjsIM44JZFFzCuNrJHWSMv7I+tD3UTIGLi4n4MT32TI+mxvqnkU7Ynx0FU86NVOIrFcOB+50ryb9y7RzTy7gQOsvR/2u6KM0lRUNLJ2hPWOAoRx4wboXXSvmAWcbR+zU36TGKi1JcOWZufaLG1WP6kyllZKiG83VM9o3fMFD/0IIvNmQOcu+q6XkD/Umi3wdYBE+njeiscfpwXr++mR1nnRySI3JCInJBrskdaZIW4WRI3skH+Qw2gyi4DK6mpUFl1nNA5iK4/Qb2G8nw</latexit> <latexit sha1_base64="HnYVzZlMAdu9BpYuDyF1qsN8nBg=">AAACjXicdVHJSgNBEO2Me9yighcvjUHwFGZc0IOI6EGPKkaFZAg9nUrS2MvQXSOGMT/jVX/Iv7En5mCMFjQ8Xi2v+lWSSuEwDD9LwdT0zOzc/EJ5cWl5ZbWytn7vTGY51LmRxj4mzIEUGuooUMJjaoGpRMJD8nRR5B+ewTph9B32U4gV62rREZyhp1qVzSbCC1qV34LQHeOHKtA4aFWqYS0cBp0E0QhUySiuW2ulVrNteFZ0c8mca0RhinHOLAouYVBuZg5Sxp9YFxoeaqbAxfnwAwO645k29fL+aaRD9mdHzpRzfZX4SsWw537nCvKvXCPDznGcC51mCJp/C3UySdHQwg3aFhY4yr4HjFvhd6W8xyzj6D0bm3QXxXmxXDFmTD5Rg/IO/ckUa6SoXsbrmOwar9BT/9CCTzakDjLvqml7A/1Jot8HmAT3e7Vov7Z3c1A9Ox8dZ55skW2ySyJyRM7IFbkmdcLJK3kj7+QjWA0Og5Pg9Ls0KI16NshYBJdfUcHL2g==</latexit> Perception Mumford, 1991, 1992 <latexit sha1_base64="gCCBi9rF50cg7j98DQC2XFcMc3c=">AAACinicdVHJSgNBEO2Me9yiOXppDIKnMBMFFS+igh4VjArJEHo6laSxl6G7RgxDvsWrfpJ/Y0+SgzFaUPB4tT1eJakUDsPwqxQsLC4tr6yuldc3Nre2Kzu7j85klkOTG2nsc8IcSKGhiQIlPKcWmEokPCUvV0X96RWsE0Y/4DCFWLG+Fj3BGXqqU6m2Ed7QqvwGNFhPvsKoU6mF9XAcdB5EU1Aj07jr7JQ67a7hmQKNXDLnWlGYYpwzi4JLGJXbmYOU8RfWh5aHmilwcT5WP6IHnunSnrE+NdIx+3MiZ8q5oUp8p2I4cL9rBflXrZVh7zTOhU4zBM0nh3qZpGhoYQXtCgsc5dADxq3wWikfMMs4esNmNj1EcV6IK9bMnE/UqHxAfzKFjBTV22wfk33jLwzUP7Tg8wOpg8y7arreQP+S6PcD5sFjox4d1Rv3x7WLy+lzVske2SeHJCIn5ILckjvSJJwMyTv5IJ/BRtAIzoLzSWtQms5UyUwE198iFMp1</latexit> Learning <latexit sha1_base64="s/2Ot/vLLppsJAuwoiAQXB18FH0=">AAACiHicdVHJSgNBEO2MW4xbokcvjUHwFGZUiN4kXjx4UDAqJEPo6VSSxl6G7ho1DPkUr/pN/o09MQdjtKDg8Wp7vEpSKRyG4WcpWFpeWV0rr1c2Nre2d6q13XtnMsuhzY009jFhDqTQ0EaBEh5TC0wlEh6Sp8ui/vAM1gmj73CcQqzYUIuB4Aw91avWugivaFV+DcxqoYeTXrUeNsJp0EUQzUCdzOKmVyv1un3DMwUauWTOdaIwxThnFgWXMKl0Mwcp409sCB0PNVPg4nyqfUIPPdOnA2N9aqRT9udEzpRzY5X4TsVw5H7XCvKvWifDwVmcC51mCJp/HxpkkqKhhRG0LyxwlGMPGLfCa6V8xCzj6O2a23QXxXkhrlgzdz5Rk8oh/ckUMlJUr/N9TA6NvzBS/9CCLw6kDjLvqul7A/1Lot8PWAT3x43opHF8e1q/aM2eUyb75IAckYg0yQW5IjekTTh5IW/knXwElSAMmsH5d2tQms3skbkIWl8ErsmH</latexit> <latexit sha1_base64="3dR8gvmwke7p4jBWz97cL9esC2o=">AAACmHicdVHJbhNBEG0PWzBLnHCDS4MVxAFZMwkSyc0ih4RbQLETyR5ZNe2y3Upv6q5BsUY+8zVc4Vv4m/Q4PsQxlNTS06vlVb8qnJKB0vRvI3nw8NHjJ1tPm8+ev3i53drZ7QdbeoE9YZX1lwUEVNJgjyQpvHQeQRcKL4qr4zp/8QN9kNac09xhrmFq5EQKoEiNWm+HhNfkdXUqDVnDh+95Hww/Bu0+8uzo6GAxarXTTroMvgmyFWizVZyNdhqj4diKUqMhoSCEQZY6yivwJIXCRXNYBnQgrmCKgwgNaAx5tfzLgu9FZswn1sdniC/Zux0V6BDmuoiVGmgW7udq8l+5QUmTw7ySxpWERtwKTUrFyfLaGD6WHgWpeQQgvIy7cjEDD4KifWuTzrO8qperx6zJF3rR3ON3mXoNR/p6vQ7U1EaFmf4PLcVmgwtYRlftOBoYT5LdP8Am6O93soPO/rdP7e6X1XG22Bv2jn1gGfvMuuyUnbEeE+wn+8V+sz/J66SbnCRfb0uTxqrnFVuL5PsNBpvOEQ==</latexit> Generative <latexit sha1_base64="Eb+mqLiTnV6p167A/LYvvHF2GSw=">AAAClXicdVFNaxsxEJU3aes6aeukhx4CQdS49FDMblpofCiYtITcklA7MdiL0cpjW0QfizRbbBbf8mtyTf5M/021jg+xnQ4IHm9m3ozeJKkUDsPwbynY2n7x8lX5dWVn983bd9W9/StnMsuhw400tpswB1Jo6KBACd3UAlOJhOvk5meRv/4D1gmj2zhLIVZsrMVIcIaeGlQP+whTtCr/nSEaTfuf6AmzaL7QqNk8ng+qtbARLoJugmgJamQZF4O90qA/NDxToJFL5lwvClOMc68puIR5pZ85SBm/YWPoeaiZAhfni4/Mad0zQzoy1j+NdME+7ciZcm6mEl+pGE7ceq4gn8v1Mhwdx7nQaYag+eOgUSYpGlq4QofCAkc584BxK/yulE+YZRy9dytK7SjOi+UKmZXxiZpX6vQpU6yRopqu1jE5Nn7CRP2HFnyzIXWQeVfN0BvoTxKtH2ATXB01oq+No8tvtdbJ8jhlckA+ks8kIt9Ji5yRC9IhnNySO3JPHoIPwY/gV3D6WBqUlj3vyUoE5/8AE+jNRw==</latexit> <latexit sha1_base64="gVsaBUX5qfHgiMjooL7477oUBQo=">AAAClHicdVHLSiNBFK30qBPjK86AGxEKg+JCQrcOqDALUQRXomBUSJpQXblJCuvRVN2WhCar+ZrZ6tf4N1bHLIzRCwWHcx/n1rlJKoXDMHwtBT/m5hd+lhcrS8srq2vV9V93zmSWQ4MbaexDwhxIoaGBAiU8pBaYSiTcJ4/nRf7+CawTRt/iMIVYsZ4WXcEZeqpd3WohDNCq/AqYpK1deik0Gr1Po5OT41G7Wgvr4TjoLIgmoEYmcd1eL7VbHcMzBRq5ZM41ozDFOGcWBZcwqrQyBynjj6wHTQ81U+DifPyPEd3xTId2jfVPIx2zHztyppwbqsRXKoZ99zlXkF/lmhl2j+Nc6DRD0PxdqJtJioYWptCOsMBRDj1g3Aq/K+V9ZhlHb93UpNsozovlijFT8okaVXboR6ZYI0U1mK5jsme8Ql99Qws+25A6yLyrpuMN9CeJPh9gFtwd1KPD+sHNn9rp2eQ4ZbJJtskeicgROSWX5Jo0CCf/yH/yTF6CjeBvcB5cvJcGpUnPbzIVwdUbqN/MqA==</latexit> <latexit sha1_base64="hWaMHF6oT4fv6FpLkR4h97cl65g=">AAACinicdVHLSgMxFE3Hd31Vu3QTLIKrMlMFFTeiLlxWsCq0Q8mkt20wmYTkjliGfotb/ST/xkzbhbV64cLh3Nfh3MRI4TAMv0rB0vLK6tr6Rnlza3tnt7K3/+h0Zjm0uJbaPifMgRQptFCghGdjgalEwlPyclPUn17BOqHTBxwZiBUbpKIvOENPdSvVDsIbWpU3wS80BTnuVmphPZwEXQTRDNTILJrdvVK309M8U5Ail8y5dhQajHNmUXAJ43Inc2AYf2EDaHuYMgUuzifqx/TIMz3a19ZninTC/pzImXJupBLfqRgO3e9aQf5Va2fYP49zkZoMIeXTQ/1MUtS0sIL2hAWOcuQB41Z4rZQPmWUcvWFzmx6iOC/EFWvmzidqXD6iP5lChkH1Nt/H5ED7C0P1Dy344oBxkHlXdc8b6F8S/X7AInhs1KOTeuP+tHZ1PXvOOjkgh+SYROSMXJE70iQtwsmIvJMP8hlsBY3gIrictgal2UyVzEVw+w1BhsqE</latexit> <latexit sha1_base64="bVdSfuQ/yQhIY1J3HEkZryJLH78=">AAACk3icdVFNSwMxEE3X7/pVFU96CBbBg5TdKqh4EfXgRVCwKrRLyWanbTDZLMmsWJZe/DVe9d/4b8zWHqzVgYTHm5nMy5solcKi73+WvKnpmdm5+YXy4tLyymplbf3e6sxwaHAttXmMmAUpEmigQAmPqQGmIgkP0dNFkX94BmOFTu6wn0KoWDcRHcEZOqpd2W4hvKBR+XWmOtrE+zQ4OQmGd33QrlT9mj8MOgmCEaiSUdy010rtVqx5piBBLpm1zcBPMcyZQcElDMqtzELK+BPrQtPBhCmwYT78xoDuOiamToU7CdIh+7MjZ8ravopcpWLYs79zBflXrplh5zjMRZJmCAn/HtTJJEVNC09oLAxwlH0HGDfCaaW8xwzj6Jwbe+kuCPNCXPHM2PhIDcq79CdTyEhRvYzXMdnVbkJP/UMLPtmQWsicqzp2BrqVBL8XMAnu67XgoFa/PayenY+WM0+2yA7ZIwE5ImfkityQBuHklbyRd/LhbXqn3rl3+V3qlUY9G2QsvOsv8IzL4A==</latexit> Olshausen & Field, 1996 <latexit sha1_base64="uipMGfmXr5m11UwrTxlfEYeg9HU=">AAACmHicdVFNTxsxEHW2tE1DW5L2BhdDBOJQRbspouUWgVTgBK0IICWryOudZK3Y65U9WxGtcubX9Nr+lv4bvEkOCWlHsvT05uON30SZFBZ9/2/Fe7Hx8tXr6pva5tt377fqjQ+3VueGQ5drqc19xCxIkUIXBUq4zwwwFUm4i8ZnZf7uJxgrdHqDkwxCxUapGArO0FGD+m4f4QGNKq6kTVhuIaX9A/pNgIw/0eDk5Hg6qDf9lj8Lug6CBWiSRVwPGpVBP9Y8V5Ail8zaXuBnGBbMoOASprW+k8kYH7MR9EpFpsCGxewvU7rvmJgOtXEvRTpjlzsKpqydqMhVKoaJfZ4ryX/lejkOv4aFSLMcIeVzoWEuKWpaGkNjYYCjnDjAuBFuV8oTZhhHZ9/KpJsgLMrlyjEr8pGa1vbpMlOukaF6WK1jcqSdQqL+Qwu+3pBZyJ2rOnYGupMEzw+wDm7breBzq/39qNk5XRynSnbIHjkkAflCOuSCXJMu4eSR/CK/yR9v2+t4597lvNSrLHo+kpXwfjwB2uLOdA==</latexit> Invertible Models Comon, 1994 Models Parra et al., 1995 Deco & Brauer, 1995 Frey et al., 1996 Bengio & Bengio, 2000 Bell & Sejnowski, 1997 van den Oord et al., 2016 <latexit sha1_base64="jWsTzQt+uILoaQZBEvFiNe+QPkM=">AAACinicdVFdSwJBFB23b7PSfOxlSISeZNeCil6ieqi3gixBF5kdRx2cj2XmbiiLv6XX+kn9m2Z1HzLrwoXDued+cG4UC27B978K3tr6xubW9k5xt7S3f1CuHL5YnRjKWlQLbdoRsUxwxVrAQbB2bBiRkWCv0fg2q7++MWO5Vs8wjVkoyVDxAacEHNUrV7vAJmBk+qCcDLjrm/XKNb/hzwOvgiAHNZTHY69S6HX7miaSKaCCWNsJ/BjClLiB1A0sdhPLYkLHZMg6DioimQ3T+fUzXHdMHw+0cakAz9mfHSmR1k5l5JSSwMj+rmXkX7VOAoOLMOUqToApulg0SAQGjTMrcJ8bRkFMHSDUcHcrpiNiCAVn2NKk5yBMs+OyMUvrIzkr1vFPJjsjBjlZ1hEx1G7DSP5Dc7raEFuWOFd13xnoXhL8fsAqeGk2gtNG8+msdn2TP2cbHaFjdIICdI6u0T16RC1E0RS9ow/06ZW8pnfpXS2kXiHvqaKl8O6+ATd3yn8=</latexit> Helmholtz Machine Dayan et al., 1995 <latexit sha1_base64="QR1yAf6xpkTfcfzNmQTmcSBRkMY=">AAACiXicdVHLTgIxFC3jC/EFuHTTSExckRk10bgysmGJCagJTEinXKCxnU7aOwac8Ctu9Zf8GzvAQkRvcpOTc18n50aJFBZ9/6vgbWxube8Ud0t7+weHR+VK9dHq1HDocC21eY6YBSli6KBACc+JAaYiCU/RSyOvP72CsULHbZwmECo2isVQcIaO6perPYQJGpU1Qaqxlvg265drft2fB10HwRLUyDJa/Uqh3xtoniqIkUtmbTfwEwwzZlBwCbNSL7WQMP7CRtB1MGYKbJjNxc/omWMGdKiNyxjpnP05kTFl7VRFrlMxHNvftZz8q9ZNcXgTZiJOUoSYLw4NU0lR09wJOhAGOMqpA4wb4bRSPmaGcXR+rWxqB2GWi8vXrJyP1Kx0Rn8yuYwE1WS1j8mRdhfG6h9a8PWBxELqXNUDZ6B7SfD7Aevg8aIeXNYvHq5qd/fL5xTJCTkl5yQg1+SONEmLdAgnE/JOPsint+cF3o13u2j1CsuZY7ISXuMbSSrKGA==</latexit> <latexit sha1_base64="2FSY/zmQNCvj7/oX+8dUqIukVl4=">AAACh3icdVHLTgIxFC3jC/AFunTTSEhc4YwaZelj48ZEE1ESmJBOuUBjO520d4xkwp+41X/yb+wACxG9yU1Ozn2dnBslUlj0/a+Ct7K6tr5RLJU3t7Z3divVvSerU8OhxbXUph0xC1LE0EKBEtqJAaYiCc/Ry01ef34FY4WOH3GcQKjYMBYDwRk6qlepdBHe0KjsjvGR2zHpVWp+w58GXQbBHNTIPO571UKv29c8VRAjl8zaTuAnGGbMoOASJuVuaiFh/IUNoeNgzBTYMJtKn9C6Y/p0oI3LGOmU/TmRMWXtWEWuUzEc2d+1nPyr1klx0AwzEScpQsxnhwappKhp7gPtCwMc5dgBxo1wWikfMcM4OrcWNj0GYZaLy9csnI/UpFynP5lcRoLqbbGPyaF2F0bqH1rw5YHEQupc1X1noHtJ8PsBy+DppBGcNk4ezmqX1/PnFMkBOSRHJCAX5JLcknvSIpy8knfyQT69knfsnXvNWatXmM/sk4Xwrr4B2VPJAg==</latexit> Hierarchical <latexit sha1_base64="cjtknKEs4uZ+9NCeJ2/8ZAhYpjU=">AAACknicdVFNTxsxEHWWFtK0pYH2xsVqhNRDFe1SEIUTBQ4ceqASAaRkFc06k8TCHyt7tkq0yoFf02v5Of039YYcCGlHsvT05uON32S5kp7i+E8tWnvxcn2j/qrx+s3bzXfNre1rbwsnsCOssu42A49KGuyQJIW3uUPQmcKb7O6syt/8ROelNVc0zTHVMDJyKAVQoPrNnR7hhJwuz2EKhiNxUO3PPDk6Opj1m624Hc+Dr4JkAVpsEZf9rVq/N7Ci0GhIKPC+m8Q5pSU4kkLhrNErPOYg7mCE3QANaPRpOf/FjO8GZsCH1oVniM/Zpx0laO+nOguVGmjsn+cq8l+5bkHDr2kpTV4QGvEoNCwUJ8srS/hAOhSkpgGAcDLsysUYHAgKxi1NukrSslquGrMkn+lZY5c/Zao1ctKT5TpQIxsUxvo/tBSrDbnHIrhqB8HAcJLk+QFWwfVeO/nS3vux3zo5XRynznbYR/aJJeyQnbALdsk6TLB79ov9Zg/Rh+g4+hadPZZGtUXPe7YU0fe/10nL4A==</latexit> <latexit sha1_base64="be+/UpaeHP5uJnLr3ADUHhwS2NI=">AAAChnicdVHJSgNBEO2MW4xLoh69NAbBU5hxQY9BL14EBZMIyRB6OpWksXt66K4RwzBf4lU/yr+xJ+aQRQsKHq+2R70okcKi73+XvLX1jc2t8nZlZ3dvv1o7OGxbnRoOLa6lNi8RsyBFDC0UKOElMcBUJKETvd4V9c4bGCt0/IyTBELFRrEYCs7QUf1atYfwjkZlD3oA0ub9Wt1v+NOgqyCYgTqZxWP/oNTvDTRPFcTIJbO2G/gJhhkzKLiEvNJLLSSMv7IRdB2MmQIbZlPlOT11zIAOtXEZI52y8xMZU9ZOVOQ6FcOxXa4V5F+1borDmzATcZIixPz30DCVFDUt3kAHwgBHOXGAcSOcVsrHzDCO7lkLm56DMCvEFWsWzkcqr5zSeaaQkaB6X+xjcqTdhbH6hxZ8dSCxkLqvOk/yirMkWDZgFbTPG8FF4/zpst68nZlTJsfkhJyRgFyTJrknj6RFOEnJB/kkX17Za3hX3vVvq1eazRyRhfCaPwYQyKc=</latexit> Deep Autoregressive Control as Inference <latexit sha1_base64="wVtgj26EC0xfnD3Vin2vxD5kqDA=">AAACk3icdVHLSgMxFE3Hd31VxZUugkVwVWZUUHDja+FGULAqtEPJpLdtMJmE5I5Yhm78Grf6N/6NmdqFtXohcDj3cW7OTYwUDsPwsxRMTc/Mzs0vlBeXlldWK2vr905nlkOda6ntY8IcSJFCHQVKeDQWmEokPCRPF0X+4RmsEzq9w76BWLFuKjqCM/RUq7LdRHhBq/JLAEPPMtQWuhacE88waFWqYS0cBp0E0QhUyShuWmulVrOteaYgRS6Zc40oNBjnzKLgEgblZubAMP7EutDwMGUKXJwPvzGgu55p0462/qVIh+zPjpwp5/oq8ZWKYc/9zhXkX7lGhp3jOBepyRBS/i3UySRFTQtPaFtY4Cj7HjBuhd+V8h6zjKN3bmzSXRTnxXLFmDH5RA3Ku/QnU6xhUL2M1zHZ1V6hp/6hBZ9sMA4y76puewP9SaLfB5gE9/u16KC2f3tYPT0fHWeebJEdskcickROyRW5IXXCySt5I+/kI9gMToLz4PK7NCiNejbIWATXX9xkzjs=</latexit> <latexit sha1_base64="DTJh0h88RPaRZuUubv+JcDhDVJc=">AAACi3icdVHJSgNBEO2Me9zicvPSGAQPEmZUcMGDKIJHBaNCMoSeTiVp7GXorhHjkH/xqn/k39gTczBGCwoer7bHqySVwmEYfpaCqemZ2bn5hfLi0vLKamVt/d6ZzHKocyONfUyYAyk01FGghMfUAlOJhIfk6bKoPzyDdcLoO+ynECvW1aIjOENPtSqbTYQXtCq/NMroPRqdnBwOWpVqWAuHQSdBNAJVMoqb1lqp1WwbninQyCVzrhGFKcY5syi4hEG5mTlIGX9iXWh4qJkCF+dD+QO645k27RjrUyMdsj8ncqac66vEdyqGPfe7VpB/1RoZdo7jXOg0Q9D8+1AnkxQNLbygbWGBo+x7wLgVXivlPWYZR+/Y2Ka7KM4LccWasfOJGpR36E+mkJGiehnvY7Jr/IWe+ocWfHIgdZB5V03bG+hfEv1+wCS4369FB7X928Pq+cXoOfNki2yTXRKRI3JOrskNqRNOXskbeScfwXJwEJwGZ9+tQWk0s0HGIrj6AmRwyZQ=</latexit> <latexit sha1_base64="Y6bUDpzzoS0bqdji4YcRGVolTK0=">AAACinicdVHJSgNBEO2Me9yiHr00BsFTmImCihdRDx4jGBWSIfR0KkljL0N3jSQM+Rav+kn+jT0xh8RoQcHj1fZ4laRSOAzDr1KwtLyyura+Ud7c2t7ZreztPzmTWQ5NbqSxLwlzIIWGJgqU8JJaYCqR8Jy83hb15zewThj9iKMUYsX6WvQEZ+ipTuWgjTBEq/Jbo9EaSZkbdyrVsBZOgi6CaAqqZBqNzl6p0+4aninQyCVzrhWFKcY5syi4hHG5nTlIGX9lfWh5qJkCF+cT9WN67Jku7RnrUyOdsLMTOVPOjVTiOxXDgftdK8i/aq0MexdxLnSaIWj+c6iXSYqGFlbQrrDAUY48YNwKr5XyAbOMozdsbtNjFOeFuGLN3PlEjcvHdJYpZKSohvN9TPaNvzBQ/9CCLw6kDjLvqul6A/1Lot8PWARP9Vp0Wqs/nFWvb6bPWSeH5IickIick2tyTxqkSTgZkXfyQT6DraAeXAZXP61BaTpzQOYiuPsGtBXKQA==</latexit> <latexit sha1_base64="be+/UpaeHP5uJnLr3ADUHhwS2NI=">AAAChnicdVHJSgNBEO2MW4xLoh69NAbBU5hxQY9BL14EBZMIyRB6OpWksXt66K4RwzBf4lU/yr+xJ+aQRQsKHq+2R70okcKi73+XvLX1jc2t8nZlZ3dvv1o7OGxbnRoOLa6lNi8RsyBFDC0UKOElMcBUJKETvd4V9c4bGCt0/IyTBELFRrEYCs7QUf1atYfwjkZlD3oA0ub9Wt1v+NOgqyCYgTqZxWP/oNTvDTRPFcTIJbO2G/gJhhkzKLiEvNJLLSSMv7IRdB2MmQIbZlPlOT11zIAOtXEZI52y8xMZU9ZOVOQ6FcOxXa4V5F+1borDmzATcZIixPz30DCVFDUt3kAHwgBHOXGAcSOcVsrHzDCO7lkLm56DMCvEFWsWzkcqr5zSeaaQkaB6X+xjcqTdhbH6hxZ8dSCxkLqvOk/yirMkWDZgFbTPG8FF4/zpst68nZlTJsfkhJyRgFyTJrknj6RFOEnJB/kkX17Za3hX3vVvq1eazRyRhfCaPwYQyKc=</latexit> <latexit sha1_base64="jfWkrHuERCpEnPXQcd6Eq3CYCXc=">AAACknicdVFNTxsxEHUW2qYplPBx42I1QuJQRbtA1cKJwqWHHlKJAFKyimadSWJhr1f2LEq0yoFfw7X8HP4N3pADITCSpac3H2/8JsmUdBSGj5VgZfXDx0/Vz7Uva+tfN+qbW5fO5FZgWxhl7HUCDpVMsU2SFF5nFkEnCq+Sm/Myf3WL1kmTXtAkw1jDMJUDKYA81avvdgnHZHXRAmuBI3FQze88Oj7+Me3VG2EznAVfBtEcNNg8Wr3NSq/bNyLXmJJQ4FwnCjOKC7AkhcJprZs7zEDcwBA7Hqag0cXF7BdTvueZPh8Y619KfMa+7ChAOzfRia/UQCP3OleSb+U6OQ1+xYVMs5wwFc9Cg1xxMry0hPelRUFq4gEIK/2uXIzAgiBv3MKkiyguyuXKMQvyiZ7W9vhLplwjIz1erAM1NF5hpN+hpVhuyBzm3lXT9wb6k0SvD7AMLg+a0WHz4N9R4/Rsfpwq22Xf2D6L2E92yv6wFmszwe7YPfvPHoKd4CT4HZw/lwaVec82W4jg7xPqnMvp</latexit> Predictive Coding <latexit sha1_base64="Lb0o/vAK/+oL1zIP6O8HzgLnEQQ=">AAACjHicdVHbSgMxEE3XW623qvjkS7AIPpVdLSiIUBTER4VehHYp2XTaBpPNksyKZenH+Kpf5N+YrX2wrQ4EDufMZA5nokQKi77/VfBWVtfWN4qbpa3tnd298v5By+rUcGhyLbV5jpgFKWJookAJz4kBpiIJ7ejlLtfbr2Cs0HEDxwmEig1jMRCcoaN65aMuwhsalT0IMMzwkVPkpFeu+FV/WnQZBDNQIbN67O0Xet2+5qmCGLlk1nYCP8EwYwYFlzApdVMLCeMvbAgdB2OmwIbZ1P+EnjqmTwfauBcjnbK/JzKmrB2ryHUqhiO7qOXkX1onxcFVmIk4SRFi/rNokEqKmuZh0L4wwFGOHWDcCOeV8hEzjKOLbO6nRhBmubn8m7n1kZqUTulvJreRoHqb72NyqN2GkfqHFnx5ILGQulR13wXoThIsHmAZtM6rwUX1/KlWqd/OjlMkx+SEnJGAXJI6eSCPpEk4ycg7+SCf3q5X8669m59WrzCbOSRz5d1/A/Qeyz4=</latexit> <latexit sha1_base64="1fsGlf9WV8fdkC9EcdHzNT+oS1M=">AAACiXicdVHLSgMxFE3Hd31VXboJFsFVmVFBcSW60V0Fq0I7lEx6pw3mMSR3pGXor7jVX/JvzLRdWKsXAodzXyfnJpkUDsPwqxIsLa+srq1vVDe3tnd2a3v7T87klkOLG2nsS8IcSKGhhQIlvGQWmEokPCevt2X++Q2sE0Y/4iiDWLG+FqngDD3Vre13EIZoVXGvU7CgOYy7tXrYCCdBF0E0A3Uyi2Z3r9Lt9AzPFWjkkjnXjsIM44JZFFzCuNrJHWSMv7I+tD3UTIGLi4n4MT32TI+mxvqnkU7Ynx0FU86NVOIrFcOB+50ryb9y7RzTy7gQOsvR/2u6KM0lRUNLJ2hPWOAoRx4wboXXSvmAWcbR+zU36TGKi1JcOWZufaLG1WP6kyllZKiG83VM9o3fMFD/0IIvNmQOcu+q6XkD/Umi3wdYBE+njeiscfpwXr++mR1nnRySI3JCInJBrskdaZIW4WRI3skH+Qw2gyi4DK6mpUFl1nNA5iK4/Qb2G8nw</latexit> Toussaint & Storkey, 2006 <latexit sha1_base64="xOL0vv65veGktzWX2OoPxcqdjOg=">AAAClHicdVHLSgMxFE3Hd31VBTciBIviQsqMD1RwIT7AlShYFdqhZNLbNphMhuSOWIau/Bq3+jX+jZnahbV6IXA493Fuzo0SKSz6/mfBGxufmJyaninOzs0vLJaWlu+tTg2HKtdSm8eIWZAihioKlPCYGGAqkvAQPZ3n+YdnMFbo+A67CYSKtWPREpyhoxql9TrCCxqVXQDXtL5FzwxLwezQ4Pj4oNcolf2K3w86CoIBKJNB3DSWCo16U/NUQYxcMmtrgZ9gmDGDgkvoFeuphYTxJ9aGmoMxU2DDrP+PHt10TJO2tHEvRtpnf3ZkTFnbVZGrVAw79ncuJ//K1VJsHYWZiJMUIebfQq1UUtQ0N4U2hQGOsusA40a4XSnvMMM4OuuGJt0FYZYvl48Zko9Ur7hJfzL5Ggmql+E6JtvaKXTUP7Tgow2JhdS5qpvOQHeS4PcBRsH9biXYq+ze7pdPzwbHmSZrZINsk4AcklNyRW5IlXDySt7IO/nwVr0T79y7/C71CoOeFTIU3vUXeDvMkQ==</latexit> <latexit sha1_base64="F+HDgiounr8VwvvbX79Zp4j5Y2k=">AAACkXicdVHLSgMxFE3Hd31V3ekmWAQXUmZU1O6KgghuFKwK7VAy6W0bTCZDckcsQ8Gvcau/49+YqbOwVi8EDuc+zs25USKFRd//LHkzs3PzC4tL5eWV1bX1ysbmvdWp4dDkWmrzGDELUsTQRIESHhMDTEUSHqKnizz/8AzGCh3f4TCBULF+LHqCM3RUp7LdRnhBo7JLA0MKSJmsHdCgXj8ZdSpVv+aPg06DoABVUsRNZ6PUaXc1TxXEyCWzthX4CYYZMyi4hFG5nVpIGH9ifWg5GDMFNszGnxjRPcd0aU8b92KkY/ZnR8aUtUMVuUrFcGB/53Lyr1wrxd5ZmIk4SRFi/i3USyVFTXNHaFcY4CiHDjBuhNuV8gEzjKPzbWLSXRBm+XL5mAn5SI3Ke/Qnk6+RoHqZrGOyr53CQP1DCz7dkFhInau66wx0Jwl+H2Aa3B/WgqPa4e1xtXFeHGeR7JBdsk8Cckoa5IrckCbh5JW8kXfy4W15da/hFbVeqejZIhPhXX8B9cTLgA==</latexit> <latexit sha1_base64="j90RZquCLRJM3MdwD7TydOvl1Oc=">AAACkXicdVHLSiNBFK30jBrjKxl3zqYwCK5CtzOg7oJuhNlkwCRC0oTq6puksB5N1W0xNAG/xu3M7/g3U53phTEzFwoO5z7OrXOTTAqHYfhWCz593treqe829vYPDo+arS8DZ3LLoc+NNPYhYQ6k0NBHgRIeMgtMJRKGyeNtmR8+gXXC6HtcZBArNtNiKjhDT02aJ2OEZ7Sq6FlIBUfxBPTWpELPlpNmO+yEq6CbIKpAm1TRm7Rqk3FqeK5AI5fMuVEUZhgXzKLgEpaNce4gY/yRzWDkoWYKXFysPrGkZ55J6dRY/zTSFfu+o2DKuYVKfKViOHcfcyX5r9wox+lVXAid5Qia/xWa5pKioaUjNBUWOMqFB4xb4XelfM4s4+h9W5t0H8VFuVw5Zk0+UcvGGX3PlGtkqJ7X65icGa8wV/+hBd9syBzk3lWTegP9SaKPB9gEg4tO9K1z8fN7u3tTHadOvpJTck4ickm65I70SJ9w8kJeyS/yOzgOroNuUNUGtarnmKxF8OMPhtLNNA==</latexit> <latexit sha1_base64="hCOmi49X+gxz7QqBXqYBtYDMx3U=">AAAClnicdVHLSiNBFK20jo84atSN4MLC4DALCd1RcFYiyqDLCEaFpAnVlZuksB5N1e3B0GTp17jVj/FvpjrJwhi9UHA493FunZukUjgMw/dSsLD4Y2l5ZbW89nN9Y7OytX3nTGY5NLmRxj4kzIEUGpooUMJDaoGpRMJ98nhZ5O//gXXC6FscphAr1teiJzhDT3Uq+22EJ7QqvwDdF4a2f9EJOqL1MAxHnUo1rIXjoPMgmoIqmUajs1XqtLuGZwo0csmca0VhinHOLAouYVRuZw5Sxh9ZH1oeaqbAxfn4JyN66Jku7Rnrn0Y6Zj925Ew5N1SJr1QMB+5zriC/yrUy7P2Jc6HTDEHziVAvkxQNLWyhXWGBoxx6wLgVflfKB8wyjt68mUm3UZwXyxVjZuQTNSof0o9MsUaK6mm2jsm+8QoD9Q0t+HxD6iDzrpquN9CfJPp8gHlwV69Fx7X6zUn1/GJ6nBWyRw7IbxKRU3JOrkmDNAknz+SFvJK3YDc4C/4GV5PSoDTt2SEzETT+A1TQzVs=</latexit> <latexit sha1_base64="vJhuMgyhHYlUzgM9jISYtuzhtW4=">AAACjXicdVHLSgMxFE3Hd31VBTdugkVwIWWmKroQEV3oUqFVoR1KJnPbBpPJkNwRy9ifcas/5N+YqV1YqxcCh3Mf5+bcKJXCou9/lryZ2bn5hcWl8vLK6tp6ZWPz3urMcGhyLbV5jJgFKRJookAJj6kBpiIJD9HTVZF/eAZjhU4aOEghVKyXiK7gDB3VqWy3EV7QqLyhY2308wGt+/7psFOp+jV/FHQaBGNQJeO47WyUOu1Y80xBglwya1uBn2KYM4OCSxiW25mFlPEn1oOWgwlTYMN89IEh3XNMTLvauJcgHbE/O3KmrB2oyFUqhn37O1eQf+VaGXZPw1wkaYaQ8G+hbiYpalq4QWNhgKMcOMC4EW5XyvvMMI7Os4lJjSDMi+WKMRPykRqW9+hPplgjRfUyWcdkTzuFvvqHFny6IbWQOVd17Ax0Jwl+H2Aa3NdrwWGtfndUvbgcH2eR7JBdsk8CckIuyA25JU3CySt5I+/kw1v3jr0z7/y71CuNe7bIRHjXX5ofyow=</latexit> <latexit sha1_base64="hzxYQ0RX0fOm3Vip7sNh38K/01M=">AAACmXicdVFNT9tAEN24LaWBQmiPXFZESBxQZAfU9kjbC+qlVCWAlFjReD1JVuyHtTtGRFbu/Bqu7V/h37AOORDSjrTS05uPN/smK5T0FMcPjejV6zdrb9ffNTc2329tt3Y+XHhbOoE9YZV1Vxl4VNJgjyQpvCocgs4UXmbX3+v85Q06L605p2mBqYaxkSMpgAI1bO0NCG/J6eoGDM/R8J/W5RyJg+oc8m6cfJoNW+24E8+Dr4JkAdpsEWfDncZwkFtRajQkFHjfT+KC0gocSaFw1hyUHgsQ1zDGfoAGNPq0mn9mxvcDk/ORdeEZ4nP2eUcF2vupzkKlBpr4l7ma/FeuX9LoS1pJU5SERjwJjUrFyfLaGZ5Lh4LUNAAQToZduZiAA0HBv6VJ50la1cvVY5bkMz1r7vPnTL1GQfp2uQ7U2AaFif4PLcVqQ+GxDK7aPBgYTpK8PMAquOh2kqNO99dx++Tb4jjrbJftsQOWsM/shJ2yM9Zjgt2xe/aH/Y12o6/RafTjqTRqLHo+sqWIfj8C/+rOeg==</latexit> <latexit sha1_base64="BbCRdmPFgawyeodlTjXEBi7ETn8=">AAAClHicdVFdSxtBFJ1srdW01diCL0UYGix9kLBrCzXggyiCT8UWo0KyhLuzN8ngzM4yc1cMS578NX3VX+O/cTbmwRh7YeBw7se5c26SK+koDB9qwZult8vvVlbr7z98XFtvbHw6d6awAjvCKGMvE3CoZIYdkqTwMrcIOlF4kVwdVfmLa7ROmuyMxjnGGoaZHEgB5Kl+Y6tHeENWl3/B8N43fghKgU13eNRutyf9RjNshdPgiyCagSabxWl/o9bvpUYUGjMSCpzrRmFOcQmWpFA4qfcKhzmIKxhi18MMNLq4nP5jwrc9k/KBsf5lxKfs844StHNjnfhKDTRyL3MV+VquW9BgLy5llheEmXgSGhSKk+GVKTyVFgWpsQcgrPS7cjECC4K8dXOTzqK4rJarxszJJ3pS3+bPmWqNnPTNfB2oofEKI/0fWorFhtxh4V01qTfQnyR6eYBFcL7bin60dv/8bB4czo6zwr6wr+w7i9gvdsBO2CnrMMFu2T92x+6DzWA/OAqOn0qD2qznM5uL4Pcjb0rMjQ==</latexit> Theoretical Neuroscience Todorov, 2008 <latexit sha1_base64="lLXGjgoxUMQknawCBoRTXl02c2M=">AAACmnicdVFNSwMxEE3X7/pV9aiHYFE8SNmtoh5FL4oXxVaFdinZdNqGJpslmRXL0h/gr/GqP8V/Y7b2YK0OBB5v3sy8zESJFBZ9/7PgzczOzS8sLhWXV1bX1ksbmw9Wp4ZDnWupzVPELEgRQx0FSnhKDDAVSXiM+pd5/vEZjBU6ruEggVCxbiw6gjN0VKtUbiK8oFFZTafWMhEjbe7Te9SmD4NDWvX9k6FT+RV/FHQaBGNQJuO4bW0UWs225qmCGLlk1jYCP8EwYwYFlzAsNlMLCeN91oWGgzFTYMNs9Jsh3XNMm3a0cc/ZGbE/KzKmrB2oyCkVw579ncvJv3KNFDtnYSbiJEWI+fegTiopapqvhraFAY5y4ADjRjivlPeYYRzdAic61YIwy83lbSbGR2pY3KM/mdxGguplUsdkV7sJPfUPLfh0QWIhdVvVbbdAd5Lg9wGmwUO1EhxVqnfH5fOL8XEWyTbZJQckIKfknFyRW1InnLySN/JOPrwd78K79m6+pV5hXLNFJsKrfQGAVs+c</latexit> <latexit sha1_base64="Od4ZHSo9vR05mHBqx3DF6eSosG4=">AAACl3icdVFNTxsxEHW2UGigJZRT1YtFBOKAol1AAk5FVKpyDCIBpGQVeZ1JYuKPlT1LiVa58mu40v/Sf1NvyCEhZSRLT28+3vhNkkrhMAz/loIPK6sf19Y/lTc2P3/Zqmx/vXEmsxxa3Ehj7xLmQAoNLRQo4S61wFQi4TYZ/Szytw9gnTC6ieMUYsUGWvQFZ+ipboV2EB7RqvwSpKSdfXoN99r8diNxSKPz89NJt1INa+E06DKIZqBKZtHobpe6nZ7hmQKNXDLn2lGYYpwzi4JLmJQ7mYOU8REbQNtDzRS4OJ9+ZUL3PNOjfWP900in7HxHzpRzY5X4SsVw6N7mCvJ/uXaG/bM4FzrNEDR/FepnkqKhhS+0JyxwlGMPGLfC70r5kFnG0bu3MKkZxXmxXDFmQT5Rk/IenWeKNVJUj4t1TA6MVxiqd2jBlxtSB5l31fS8gf4k0dsDLIObo1p0XDu6OqleXM6Os06+k11yQCJySi5InTRIi3DyRJ7JC/kTfAt+BL+C+mtpUJr17JCFCK7+AeqZzhE=</latexit> Rao & Ballard, 1999 Machine Learning Control Theory <latexit sha1_base64="2FSY/zmQNCvj7/oX+8dUqIukVl4=">AAACh3icdVHLTgIxFC3jC/AFunTTSEhc4YwaZelj48ZEE1ESmJBOuUBjO520d4xkwp+41X/yb+wACxG9yU1Ozn2dnBslUlj0/a+Ct7K6tr5RLJU3t7Z3divVvSerU8OhxbXUph0xC1LE0EKBEtqJAaYiCc/Ry01ef34FY4WOH3GcQKjYMBYDwRk6qlepdBHe0KjsjvGR2zHpVWp+w58GXQbBHNTIPO571UKv29c8VRAjl8zaTuAnGGbMoOASJuVuaiFh/IUNoeNgzBTYMJtKn9C6Y/p0oI3LGOmU/TmRMWXtWEWuUzEc2d+1nPyr1klx0AwzEScpQsxnhwappKhp7gPtCwMc5dgBxo1wWikfMcM4OrcWNj0GYZaLy9csnI/UpFynP5lcRoLqbbGPyaF2F0bqH1rw5YHEQupc1X1noHtJ8PsBy+DppBGcNk4ezmqX1/PnFMkBOSRHJCAX5JLcknvSIpy8knfyQT69knfsnXvNWatXmM/sk4Xwrr4B2VPJAg==</latexit> <latexit sha1_base64="OXMBzYqb1Qk9MiFTsQ4m7xQ0e3k=">AAACi3icdVHJSgNBEO2Me9wS9ealMQiewowRFPEgiuBRIYuQDKGnU0kau6eH7hpJHPIvXvWP/Bt7Yg7GaEHB471aHlVRIoVF3/8seEvLK6tr6xvFza3tnd1Sea9pdWo4NLiW2jxFzIIUMTRQoISnxABTkYRW9Hyb660XMFbouI7jBELFBrHoC87QUd3SQQdhhEZl9SFoA+gEOemWKn7VnwZdBMEMVMgsHrrlQrfT0zxVECOXzNp24CcYZsy4gRImxU5qIWH8mQ2g7WDMFNgwm9qf0GPH9GhfG5cx0in7syNjytqxilylYji0v7Wc/Etrp9i/CDMRJylCzL8X9VNJUdP8FrQnDHCUYwcYN8J5pXzIDOPoLjY3qR6EWW4uHzO3PlKT4jH9yeQ2ElSj+TomB9ptGKp/aMEXGxILqbuq7rkDupcEvx+wCJqn1aBWPX08q1zfzJ6zTg7JETkhATkn1+SePJAG4eSVvJF38uFtezXv0rv6LvUKs559Mhfe3RcrwMrp</latexit> <latexit sha1_base64="EQCN1geucAdf2sTINZWHMkTu1kk=">AAACjHicdVHLSgMxFE3HV62vqrhyEyyCqzKjgoIIRUFcSQWrQjuUTHrbBpPJkNwRy9CPcatf5N+YabvoQy8EDue+Ts6NEiks+v5PwVtaXlldK66XNja3tnfKu3vPVqeGQ4Nrqc1rxCxIEUMDBUp4TQwwFUl4id5u8/zLOxgrdPyEgwRCxXqx6ArO0FHt8kEL4QONyh4gNdpyATGHYbtc8av+KOgiCCagQiZRb+8W2q2O5qmCGLlk1jYDP8EwYwYFlzAstVILCeNvrAdNB2OmwIbZSP+QHjumQ7vauBcjHbHTHRlT1g5U5CoVw76dz+XkX7lmit3LMBNxkqL713hRN5UUNc3NoB1hgKMcOMC4EU4r5X1mGEdn2cykpyDMcnH5mJn1kRqWjuk0k8tIUH3M1jHZ025DX/1DC77YkFhInau64wx0JwnmD7AInk+rwVn19PG8UruZHKdIDskROSEBuSA1ck/qpEE4ycgn+SLf3rZ37l151+NSrzDp2Scz4d39Aj/7y2I=</latexit> <latexit sha1_base64="5k3xzP/fYxkjDiIboJ6NPBbROYc=">AAACh3icdVHJTgJBEG3GHTfUo5eOxMQTzqhRjy4Xj5oImMCE9DQFdOxl0l1DJBP+xKv+k39jD3AQ0UoqeXm1vbxKUikchuFXKVhaXlldW98ob25t7+xW9vYbzmSWQ50baexLwhxIoaGOAiW8pBaYSiQ0k9f7ot4cgnXC6GccpRAr1teiJzhDT3UqlTbCG1qV3xuN1shxp1INa+Ek6CKIZqBKZvHY2St12l3DMwUauWTOtaIwxThnFgWXMC63Mwcp46+sDy0PNVPg4nwifUyPPdOlPWN9aqQT9udEzpRzI5X4TsVw4H7XCvKvWivD3nWcC51mCJpPD/UySdHQwgfaFRY4ypEHjFvhtVI+YJZx9G7NbXqO4rwQV6yZO5+ocfmY/mQKGSmqt/k+JvvGXxiof2jBFwdSB5l31XS9gf4l0e8HLILGWS06r509XVRv7mbPWSeH5IickIhckRvyQB5JnXAyJO/kg3wGG8FpcBlcT1uD0mzmgMxFcPsNNRPJLg==</latexit> <latexit sha1_base64="TSFvy/724+7QsBcNjuuSew7v/DU=">AAAChnicdVHJTgJBEG3GDXEB9eilIzHxRGZcgkeiF4+aAJrAhPQ0BXTsnp501xjIhC/xqh/l39gDHFi0kkpeXm0vr6JECou+/1PwtrZ3dveK+6WDw6PjcuXktG11aji0uJbavEXMghQxtFCghLfEAFORhNfo/TGvv36AsULHTZwkECo2jMVAcIaO6lXKXYQxGpU1R6DNZNqrVP2aPwu6CYIFqJJFPPdOCr1uX/NUQYxcMms7gZ9gmDGDgkuYlrqphYTxdzaEjoMxU2DDbKZ8Si8d06cDbVzGSGfs8kTGlLUTFblOxXBk12s5+Vetk+LgPsxEnKQIMZ8fGqSSoqa5DbQvDHCUEwcYN8JppXzEDOPozFrZ1AzCLBeXr1k5H6lp6ZIuM7mMBNV4tY/JoXYXRuofWvDNgcRC6lzVfWege0mw/oBN0L6uBTe165fbauNh8ZwiOScX5IoEpE4a5Ik8kxbhJCWf5It8e0Wv5t159XmrV1jMnJGV8Bq/NenIvg==</latexit> <latexit sha1_base64="s/2Ot/vLLppsJAuwoiAQXB18FH0=">AAACiHicdVHJSgNBEO2MW4xbokcvjUHwFGZUiN4kXjx4UDAqJEPo6VSSxl6G7ho1DPkUr/pN/o09MQdjtKDg8Wp7vEpSKRyG4WcpWFpeWV0rr1c2Nre2d6q13XtnMsuhzY009jFhDqTQ0EaBEh5TC0wlEh6Sp8ui/vAM1gmj73CcQqzYUIuB4Aw91avWugivaFV+DcxqoYeTXrUeNsJp0EUQzUCdzOKmVyv1un3DMwUauWTOdaIwxThnFgWXMKl0Mwcp409sCB0PNVPg4nyqfUIPPdOnA2N9aqRT9udEzpRzY5X4TsVw5H7XCvKvWifDwVmcC51mCJp/HxpkkqKhhRG0LyxwlGMPGLfCa6V8xCzj6O2a23QXxXkhrlgzdz5Rk8oh/ckUMlJUr/N9TA6NvzBS/9CCLw6kDjLvqul7A/1Lot8PWAT3x43opHF8e1q/aM2eUyb75IAckYg0yQW5IjekTTh5IW/knXwElSAMmsH5d2tQms3skbkIWl8ErsmH</latexit> Free Energy & Active Inference <latexit sha1_base64="v3hNAGhdfkr6EV77mvCkhXlE65E=">AAACjnicdVHLattAFB0rbZO6LyVddJHNUOPSlZHc0mRTGhpaskzBjg22MKPRtT14HmLmKsQIfU237Qf1bzJyvIji9MLA4dzHuXNumkvhMIr+tYK9J0+f7R88b794+er1m/Dw6MqZwnIYciONHafMgRQahihQwji3wFQqYZSuzuv86BqsE0YPcJ1DothCi7ngDD01C99NEW7QqvKnBaA/NNjFmk4/VLOwE/WiTdBdEG9Bh2zjcnbYmk0zwwsFGrlkzk3iKMekZBYFl1C1p4WDnPEVW8DEQ80UuKTc/KCiXc9kdG6sfxrphr3fUTLl3FqlvlIxXLqHuZp8LDcpcH6alELnBYLmd0LzQlI0tLaDZsICR7n2gHEr/K6UL5llHL1pjUmDOCnr5eoxDflUVe0uvc/Ua+Sobpp1TC6MV1iq/9CC7zbkDgrvqsm8gf4k8cMD7IKrfi/+1Ov/+tw5+749zgE5Ju/JRxKTE3JGLsglGRJOKvKb/CF/gzD4EnwNvt2VBq1tz1vSiODiFil1y0E=</latexit> Deep RL <latexit sha1_base64="YoauQic1jVCgg8B4Oc+GBod8I8M=">AAACkHicdVHJbhpBEG3GWRyyGPApyqUVZCknNONEMreAc3FyIhJgSzBCPU0NtNzLqLsGgUbIX+Nr8j35m/RgDixJSSU9vdpfJZkUDsPwTyU4efb8xcvTV9XXb96+O6vVG0NncsthwI009i5hDqTQMECBEu4yC0wlEm6T+29l/HYB1gmj+7jKIFZspkUqOENPTWrvxwhLtKrochQLoN91ChY0h/Wk1gxb4cboMYi2oEm21pvUK5Px1PBcgUYumXOjKMwwLphFwSWsq+PcQcb4PZvByEPNFLi42NywpheemdLUWO8a6YbdrSiYcm6lEp+pGM7dYawk/xUb5Zi240LoLEd/19OgNJcUDS0FoVNhgaNcecC4FX5XyufMMo5etr1O/SguyuXKNnvjE7WuXtBdplwjQ7Xcz2NyZvyEufoPLfhxQeYg96qaqRfQvyQ6fMAxGF62os+ty59fmp3r7XNOyQfykXwiEbkiHXJDemRAOHkgj+QX+R00gnbwNeg+pQaVbc052bPgx19cUcyy</latexit> Variational Autoencoder Friston, 2005, 2009 <latexit sha1_base64="qjZxr/WGwtwxRYiBzdovmzxlphY=">AAACh3icdVHLSgMxFE3Hd31VXboJFsFVnVFRl74WLlyoWBXaoWTS2zY0mYTkjliG/olb/Sf/xkztwlq9cOFw7utwbmKkcBiGn6VgZnZufmFxqby8srq2XtnYfHQ6sxzqXEttnxPmQIoU6ihQwrOxwFQi4SnpXxb1pxewTuj0AQcGYsW6qegIztBTrUqlifCKVuVXAIbe3wxblWpYC0dBp0E0BlUyjtvWRqnVbGueKUiRS+ZcIwoNxjmzKLiEYbmZOTCM91kXGh6mTIGL85H0Id31TJt2tPWZIh2xPydyppwbqMR3KoY997tWkH/VGhl2TuNcpCZDSPn3oU4mKWpa+EDbwgJHOfCAcSu8Vsp7zDKO3q2JTQ9RnBfiijUT5xM1LO/Sn0whw6B6nexjsqv9hZ76hxZ8esA4yLyruu0N9C+Jfj9gGjwe1KLD2sHdUfXsYvycRbJNdsgeicgJOSPX5JbUCScv5I28k49gKdgPjoPT79agNJ7ZIhMRnH8B3ZnIiQ==</latexit> <latexit sha1_base64="Rmp7t8UJ9vZQM5Yl0fY1Z15t+ys=">AAACi3icdVHLSitBEO2Mj6u5PuJj56YxCHcVZlTwcnEhiuBSwUQhGUJNp5I0dk8P3TUX45B/cat/5N/YE7MwRgsKDqdeh1NJpqSjMHyrBAuLS8u/Vlarv9fWNzZrW9stZ3IrsCmMMvY+AYdKptgkSQrvM4ugE4V3ycNFWb/7j9ZJk97SKMNYwyCVfSmAPNWt7XYIH8nqogVWTjhQ426tHjbCSfB5EE1BnU3jurtV6XZ6RuQaUxIKnGtHYUZxAZakUDiudnKHGYgHGGDbwxQ0uriYyB/zA8/0eN9YnynxCft5ogDt3EgnvlMDDd3XWkl+V2vn1P8bFzLNcsJUfBzq54qT4aUXvCctClIjD0BY6bVyMQQLgrxjM5tuo7goxZVrZs4nelw94J+ZUkZG+nG2D9TA+AtD/QMtxfxA5jD3rpqeN9C/JPr6gHnQOmxER43Dm+P62fn0OStsj+2zPyxiJ+yMXbFr1mSCPbFn9sJeg/XgKPgXnH60BpXpzA6bieDyHTgmyu8=</latexit> <latexit sha1_base64="+QxJthZA1r2tptAyMrnZHcjCV24=">AAACk3icdVFNSyNBEO2MH6tZP6LiSQ/NBmEPS5iJLrviRVTEi6BgVEiG0NOpJI39MXTXiGHIxV/jVf+N/8aemIMxWtDN41VVv+pXSSqFwzB8LQUzs3PzPxYWyz+XlldWK2vr185klkODG2nsbcIcSKGhgQIl3KYWmEok3CR3x0X+5h6sE0Zf4SCFWLGeFl3BGXqqXdluITygVfmp9WJG/6H1MPw7uveH7Uo1rIWjoNMgGoMqGcdFe63UbnUMzxRo5JI514zCFOOcWRRcwrDcyhykjN+xHjQ91EyBi/PRN4Z0xzMd2jXWH410xH7syJlybqASX6kY9t3nXEF+lWtm2P0f50KnGYLm70LdTFI0tPCEdoQFjnLgAeNW+Fkp7zPLOHrnJl66iuK8GK54ZkI+UcPyDv3IFGOkqB4m65jsGa/QV9/Qgk83pA4y76rpeAP9SqLPC5gG1/VatFurX+5VD4/Gy1kgW+QX+U0i8o8ckjNyQRqEk0fyRJ7JS7AZHARHwcl7aVAa92yQiQjO3wDXk8vU</latexit> Flow-Based Models / Normalizing Flows <latexit sha1_base64="4OcZ8wQaq9D4k13vKn4AOWL3c0g=">AAACi3icdVHLSgMxFE3Hd33Vx85NsAiuyowKirioiuBSwarQDiWT3rahyWRI7kjrMP/iVv/IvzFTu7BWLwQO575O7okSKSz6/mfJm5tfWFxaXimvrq1vbFa2th+tTg2HBtdSm+eIWZAihgYKlPCcGGAqkvAUDa6L/NMLGCt0/ICjBELFerHoCs7QUe3KbgthiEZllylqiLnugMnblapf88dBZ0EwAVUyibv2Vqnd6mieKoiRS2ZtM/ATDDNmUHAJebmVWkgYH7AeNB2MmQIbZmP5OT1wTId2tXEvRjpmf3ZkTFk7UpGrVAz79neuIP/KNVPsnoWZiJMU3de+F3VTSVHT4ha0IwxwlCMHGDfCaaW8zwzj6C42NekhCLNCXDFman2k8vIB/ckUMhJUw+k6Jnvabeirf2jBZxsSC6m7qrMkLztLgt8GzILHo1pwXDu6P6nWrybmLJM9sk8OSUBOSZ3ckjvSIJy8kjfyTj68de/YO/cuvku90qRnh0yFd/MFNgHK7g==</latexit> Kingma & Welling, 2014 Rezende et al., 2014 <latexit sha1_base64="8DLKz9c6IAIMXyxA/XZlKJ7RVm4=">AAAClHicdVHLSiNBFK30+Iyv6IAbEQqD4EJCdxRmFrOQkQFXomJUSJpQXX2TFKlHU3V7MDZZ+TVuZ75m/maqYxbG6IWCw7mPc+vcJJPCYRj+qwRfFhaXlldWq2vrG5tbte2dO2dyy6HFjTT2IWEOpNDQQoESHjILTCUS7pPheZm//w3WCaNvcZRBrFhfi57gDD3Vre13EB7RquIGnkCnQAEpk41j2gyj03G3Vg8b4SToPIimoE6mcdXdrnQ7qeG5Ao1cMufaUZhhXDCLgksYVzu5g4zxIetD20PNFLi4mPxjTA89k9Kesf5ppBP2bUfBlHMjlfhKxXDg3udK8qNcO8fe97gQOssRNH8V6uWSoqGlKTQVFjjKkQeMW+F3pXzALOPorZuZdBvFRblcOWZGPlHj6iF9y5RrZKgeZ+uY7BuvMFCf0ILPN2QOcu+qSb2B/iTR+wPMg7tmIzppNK9P62c/p8dZIXvkgByRiHwjZ+SCXJEW4eSZvJA/5G+wG/wIzoNfr6VBZdrzlcxEcPkf50XMww==</latexit> <latexit sha1_base64="I/QuPtvRfZxSNynElhkN2rR2l5w=">AAACkXicdVFdSxtBFJ2sbbWxrVHf6svQIPhQwm4UtG9BXwQpKBgTSJYwO7lJBudjmblbDEvAX+Or/h3/jbPpPuRDLwwczv04d85NUikchuFrJdj49PnL5tbX6va37z92art7d85klkObG2lsN2EOpNDQRoESuqkFphIJneT+osh3/oF1wuhbnKYQKzbWYi R <latexit sha1_base64="TBfYbGPVwodTvBLsayecWZ/AQXI=">AAACkXicdVFbSxtBFJ5srZdYbaxv+jI0CD6FXRWqb0FBCoWikBghWcLs5CQZnMsyc1aNS8Bf09f27/TfdDbuQy72wMDHdy7fme8kqRQOw/BvJfiw9nF9Y3Oruv1pZ/dzbe/LnTOZ5dDmRhp7nzAHUmhoo0AJ96kFphIJneThqsh3HsE6YXQLJynEio20GArO0FP92kEP4Rmtyn8aq5gUL0KP6LU0T27ar9XDRjgLugqiEtRJGTf9vUq/NzA8U6CRS+ZcNwpTjHNmUXAJ02ovc5Ay/sBG0PVQMwUuzmefmNIjzwzo0Fj/NNIZO9+RM+XcRCW+UjEcu+VcQb6X62Y4PI9zodMMQfM3oWEmKRpaOEIHwgJHOfGAcSv8rpSPmWUcvW8Lk1pRnBfLFWMW5BM1rR7ReaZYI0X1vFjH5Mh4hbH6Dy34akPqIPOumoE30J8kWj7AKrg7aUSnjZPbs3rzsjzOJjkkX8kxicg30iTfyQ1pE05eyS/ym/wJ9oOLoBmUtUGl7NknCxH8+Afv781m</latexit> Dinh et al., 2015, 2017 <latexit sha1_base64="LkGr7SoYiSXqFM29iu0g/wqEQeU=">AAACl3icdVHLTiJBFC16HjI4zuC4MrOpDDFxYUg3o9GdRo1xCQmgCXRIdXGBivXoVN02kA5bv8at/ot/YzWyEJm5ya2cnPusc5NUCodh+FIKPn3+8nWj/K2y+X3rx8/q9q+uM5nl0OFGGnubMAdSaOigQAm3qQWmEgk3yd1FEb+5B+uE0W2cpRArNtZiJDhDTw2qtI8wRavyS6EnFJAyWT+gjTA6WrzH80G1FtbDhdF1EC1BjSytOdguDfpDwzMFGrlkzvWiMMU4ZxYFlzCv9DMHKeN3bAw9DzVT4OJ88ZU53fPMkI6M9a6RLtj3FTlTzs1U4jMVw4n7GCvIf8V6GY5O4lzoNEPQ/G3QKJMUDS10oUNhgaOcecC4FX5XyifMMo5evZVO7SjOi+WKNivjEzWv7NH3TLFGimq6msfk2PgJE/UfWvD1gtRB5lU1Qy+gP0n08QDroNuoR3/rjdZh7ex8eZwy+U3+kH0SkWNyRq5Jk3QIJw/kkTyR52A3OA2uguu31KC0rNkhKxa0XgEA8Myu</latexit> Part II Mnih et al., 2013 <latexit sha1_base64="pd2CMaIw/V7K12VfXivzKKKDpLA=">AAAClHicdVFdSxtBFJ2s1qaxamKhLyIMhkBfGndtoX3ogyiKL0qExASSJcxObpLBmZ1l5m5rWPapv6av9df033Q25iExemHgcO7HuXNPlEhh0ff/lbyNzTdbb8vvKtvvd3b3qrX9O6tTw6HDtdSmFzELUsTQQYESeokBpiIJ3ej+vMh3f4KxQsdtnCUQKjaJxVhwho4aVg8HCA9oVHYp9a/PZ27SiF7rEUhLj2k+rNb9pj8Pug6CBaiTRbSGtdJwMNI8VRAjl8zafuAnGGbMoOAS8sogtZAwfs8m0HcwZgpsmM3/kdNGWsiPtXEvRjpnlzsypqydqchVKoZT+zxXkC/l+imOv4eZiJMUIeZPQuNUUtS0OAodCQMc5cwBxo1wu1I+ZYZxdKdbmdQOwqxYrhizIh+pvNKgy0yxRoLqYbWOyYl2ClP1Ci34ekNiIXVXdbbkFWdJ8NyAdXB30gy+NE9uv9ZPzxbmlMkBOSKfSEC+kVNyRVqkQzj5Tf6Qv+TR++j98M69i6dSr7To+UBWwrv5DzqHzWY=</latexit> <latexit sha1_base64="HZUo/TNwsx8j1jppcADLd0oV26E=">AAACl3icdVFNSyNBEO2M33F1o57ES2Nw8SBhJiu4N0VBBC8KxgjJEHo6lUljfwzdNYthyNVf41X/y/6b7Yk5GKMFDY9XVa+qXyWZFA7D8F8lWFhcWl5ZXauu/9jY/Fnb2r53JrccWtxIYx8S5kAKDS0UKOEhs8BUIqGdPF6U+fZfsE4YfYejDGLFUi0GgjP0VK9GuwhPaFVxLXSqGO3+om2QXi09os0wOh73avWwEU6CzoNoCupkGje9rUqv2zc8V6CRS+ZcJwozjAtmUXAJ42o3d5Ax/shS6HiomQIXF5OvjOmBZ/p0YKx/GumE/dhRMOXcSCW+UjEcus+5kvwq18lx8CcuhM5yBM3fBw1ySdHQ0hfaFxY4ypEHjFvhd6V8yCzj6N2bUbqL4qJcrpSZGZ+ocfWAfmTKNTJUT7N1TKbGTxiqb2jB5xsyB7l31fS9gf4k0ecDzIP7ZiP63WjeHtfPzqfHWSV7ZJ8ckoickDNyRW5Ii3DyTF7IK3kLdoPT4DK4ei8NKtOeHTITwe1/nXDN6w==</latexit> A Part III m L Figure 1 1: Conceptual Overview The concepts put forth by cybernetics permeated the modern fields of theoretical neuroscience (left), machine learning (center), and control theory (right). The graph attempts to identify key concepts and several relevant works in these areas although the graph can barely begin to capture the full breadth and depth of prior work, as well as the overlap between control theory and theoretical neuroscience At the bottom of the graph we have coarsely labeled how Parts II & III of this thesis fit into the conceptual landscape feedback) Such models have incorporated many aspects of neuroscience including local learning rules (Friston, 2005) and attention (Spratling, 2008; Feldman and Friston, 2010; Kanai et al., 2015), and have been coarsely compared with neural circuits (Bastos et al., 2012; Keller and Mrsic-Flogel, 2018; Walsh et al., 2020). While predictive coding and other forms of Bayesian brain theories have become increasingly popular (Doya et al., 2007; Friston, 2009; Clark, 2013), empirically testing these normative models remains challenging. This is partially due to the difficulty of distinguishing between the large number of specific design choices and the more general theoretical claims of probabilistic learning and inference (S. J. Gershman, 2019). Further, because these models have been limited to simplified implementations often without learned parameters it has been difficult to bridge the gap to the complexity of biological neural systems 6 The AI winter thawed in the early 2010s, brought on by advances in parallel computing as well as standardized datasets (Deng et al., 2009; Krizhevsky and Hinton, 2009) and environments (Todorov, Erez, and Tassa, 2012; Bellemare et al., 2013). In this new era of deep learning (LeCun, Y. Bengio, and Hinton, 2015; Schmidhuber, 2015), i.e., artificial neural networks with multiple layers, a flourishing of ideas emerged around probabilistic modeling. Building off of previous work, more expressive classes of deep hierarchical (Gregor et al., 2014; Mnih and Gregor, 2014; Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014), autoregressive (Uria, I. Murray, and Larochelle, 2014; Oord, Kalchbrenner, and Kavukcuoglu, 2016), and invertible (Dinh, Krueger, and Y. Bengio, 2015; Dinh, Sohl-Dickstein, and S. Bengio, 2017) probabilistic models were developed. Of particular importance is a class of models known as variational autoencoders (VAEs) (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014), which, as we discuss in Chapter 8, bear a close resemblance to hierarchical predictive coding models. Despite this similarity, the machine learning community remains largely oblivious to the progress in predictive coding that occurred during the AI winter and vice versa. Bridging this divide is one of the primary motivations for this thesis (Section 1.3). The discussion in this section, both in neuroscience and machine learning, has centered on probabilistic modeling and inference for perception. However, as noted earlier, these techniques can be equally applied to control. This connection was reestablished in the various fields that descended from cybernetics, particularly machine learning (Dayan and Hinton, 1997; Attias, 2003), control theory (Toussaint and Storkey, 2006; Todorov, 2008), and theoretical neuroscience (Friston, Daunizeau, and Kiebel, 2009; Botvinick and Toussaint, 2012). While these approaches differ in their exact formulations, they have a similar theme: “observing” desired outcomes, then using a probabilistic model to infer corresponding actions. Friston and colleagues have worked toward connecting predictive coding with control through a framework referred to as active inference (Friston, Daunizeau, and Kiebel, 2009; Friston, FitzGerald, et al., 2017). Modern machine learning, in contrast, often formulates control through a framework known as reinforcement learning (RL) as inference (Levine, 2018). Although these frameworks are not strictly equivalent (Millidge et al., 2020), they share many mathematical concepts. Machine learning and neuroscience have diverged in many ways over the past two decades, however, the fundamental ideas emphasized by cybernetics, i.e., probabilistic modeling and inference for perception and control, remain the focus of both fields. 7 It is unclear whether these fields will independently arrive at similar solutions for building and understanding, respectively, autonomous agents. However, given the similarity of ideas separately housed within each field, it seems likely that bridging these divides will help to once again facilitate cross-pollination and progress. 1.3 Predictive Coding Meets Machine Learning This thesis is, in large part, an attempt to fortify a bridge between machine learning and neuroscience by translating the ideas of predictive coding into deep probabilistic models. While the present work provides unique contributions, many of the insights were inspired by previous works at this intersection. In particular, Broeke, 2016 outlines the relationship between hierarchical probabilistic models in predictive coding and deep learning. Likewise, Lotter, Kreiman, and Cox, 2017 implement basic predictive coding techniques in deep probabilistic models, later comparing these models with neural phenomena (Lotter, Kreiman, and Cox, 2018). This thesis draws upon these ideas and many others, formalizing and extending them. Concretely, this thesis takes the negative feedback and feedforward techniques developed in predictive coding and applies them in machine learning using the mathematics of amortization (S. Gershman and Goodman, 2014) and normalizing flows (Rezende and Mohamed, 2015; Kingma, Salimans, et al., 2016), respectively. Briefly, amortization facilitates efficiently updating estimates to minimize prediction errors (Chapters 3, 4, and 6), i.e., (negative) feedback, whereas normalizing flows, in our formulation, provide a mechanism for improving predictions across time (Chapters 5 and 7), i.e., feedforward. These techniques, which are both learned from data, are demonstrated in the contexts of perception (Part II) and control (Part III). An outline of the thesis content is provided in Table 1.1. A broad conceptual overview of the inspirations for this thesis is also provided in Figure 1.1. While the construction of this conceptual bridge has proceeded unidirectionally from neuroscience toward machine learning, the insights developed in this process have striking implications in the reverse direction. In Chapter 8, we discuss these implications in depth. In short, traversing the bridge from machine learning, back through predictive coding, to neuroscience provides new perspectives regarding the correspondence between biological and artificial neural networks, as well as computational aspects of normalization in neural circuits (Carandini and Heeger, 2012). Though this discussion is more speculative in nature than the technical contributions of this thesis, it provides multiple lines of inquiry for future cross- 8 Table 1.1: Thesis Organization. Feedback Feedforward Perception (Part II) Chapters 3 & 4 Chapter 5 Control (Part III) Chapter 6 Chapter 7 pollination efforts. 1.4 The Future of Feedback & Feedforward The distance between neuroscience and machine learning is to be expected, given their differing aims and system constraints. A perfect convergence of these fields is neither essential nor necessarily beneficial. Yet, the ideas set forth by cybernetics three quarters of a century ago, namely perception and control through feedback and feedforward processing, are alive and well in both fields. This thesis works toward uniting these areas, developing the ideas of cybernetics with modern computational tools. The result is a set of basic techniques for probabilistic modeling and inference based on feedback and feedforward principles, applicable to both perception and control. However, scientific research, by its very nature, is constrained to simplified settings, where complex phenomena can be studied in isolation. The real work remains to be done in both integrating these ideas into a cohesive embodied system, as discussed in Chapter 9, as well as pushing these ideas and others to their limits to build and understand bigger, more expressive systems. If we take the correspondences in Chapter 8 seriously, biological systems still contain orders of magnitude more parameters than current machine learning models, likely with many more unforeseen complexities. If we hope to build more capable autonomous systems or understand neural processing in computational terms, a closer collaboration between these fields may prove fruitful. Indeed, these discussions are already bearing fruit: Richards and colleagues, inspired in part by the predictive coding-based machine learning models of Lotter, Kreiman, and Cox, 2017, have identified top-down predictions and bottom-up error signals in separate dendritic compartments of neocortical pyramidal neurons (Gillon et al., 2021). In this way, the exploration of predictive coding ideas in machine learning has helped neuroscientists pose more targeted scientific hypotheses. Neuroscience and machine learning may never return to the close collaboration envisaged under cybernetics. Nevertheless, a mutually beneficial dialogue can be 9 maintained by tying these fields together around their unifying concepts, such as feedback and feedforward perception and control. Considering systems in these cybernetic terms could help us move beyond the foggy notions of biological and artificial intelligence. Instead, we can understand organisms and autonomous systems from a more firmly-grounded scientific and engineering perspective, in terms of feedback and feedforward computational processing, homeostatic objectives, and ultimately their physical embodiment of complexity and information. This thesis is a step in that direction. References Ackley, David H, Geoffrey E Hinton, and Terrence J Sejnowski (1985). “A learning algorithm for Boltzmann machines”. In: Cognitive science 9.1, pp. 147–169. Ashby, W Ross (1956). An Introduction to Cybernetics. Chapman and Hall. Astrom, Karl Johan and Richard M Murray (2008). Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press. Attias, Hagai (2003). “Planning by probabilistic inference.” In: AISTATS. Citeseer. Barlow, Horace B (1961). “The coding of sensory messages”. In: Current problems in animal behavior. – (1989). “Unsupervised learning”. In: Neural computation 1.3, pp. 295–311. Bastos, Andre Moraes et al. (2012). “Canonical microcircuits for predictive coding”. In: Neuron 76.4, pp. 695–711. Bell, Anthony J and Terrence J Sejnowski (1997). “The “independent components” of natural scenes are edge filters”. In: Vision research 37.23, pp. 3327–3338. Bellemare, Marc G et al. (2013). “The arcade learning environment: An evaluation platform for general agents”. In: Journal of Artificial Intelligence Research 47, pp. 253–279. Bengio, Yoshua and Samy Bengio (2000). “Modeling high-dimensional discrete data with multi-layer neural networks”. In: Advances in Neural Information Processing Systems, pp. 400–406. Bialek, William et al. (1991). “Reading a neural code”. In: Science 252.5014, pp. 1854–1857. Botvinick, Matthew and Marc Toussaint (2012). “Planning as inference”. In: Trends in cognitive sciences 16.10, pp. 485–488. Broeke, Gerben van den (2016). “What auto-encoders could learn from brains”. MA thesis. Aalto University. Carandini, Matteo and David J Heeger (2012). “Normalization as a canonical neural computation”. In: Nature Reviews Neuroscience 13.1, pp. 51–62. 10 Clark, Andy (2013). “Whatever next? Predictive brains, situated agents, and the future of cognitive science”. In: Behavioral and Brain Sciences 36.3, pp. 181–204. Comon, Pierre (1994). “Independent component analysis, a new concept?” In: Signal processing 36.3, pp. 287–314. Conway, Flo and Jim Siegelman (2006). Dark hero of the information age: In search of Norbert Wiener, the father of cybernetics. Basic Books. Dayan, Peter and Geoffrey E Hinton (1996). “Varieties of Helmholtz machine”. In: Neural Networks 9.8, pp. 1385–1403. – (1997). “Using expectation-maximization for reinforcement learning”. In: Neural Computation 9.2, pp. 271–278. Dayan, Peter, Geoffrey E Hinton, et al. (1995). “The helmholtz machine”. In: Neural computation 7.5, pp. 889–904. Deco, Gustavo and Wilfried Brauer (1995). “Higher order statistical decorrelation without information loss”. In: Advances in Neural Information Processing Systems, pp. 247–254. Deng, Jia et al. (2009). “Imagenet: A large-scale hierarchical image database”. In: 2009 IEEE conference on computer vision and pattern recognition. Ieee, pp. 248– 255. Dinh, Laurent, David Krueger, and Yoshua Bengio (2015). “Nice: Non-linear independent components estimation”. In: International Conference on Learning Representations. Dinh, Laurent, Jascha Sohl-Dickstein, and Samy Bengio (2017). “Density estimation using real nvp”. In: International Conference on Learning Representations. Dong, Dawei W and Joseph J Atick (1995). “Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus”. In: Network: Computation in Neural Systems 6.2, pp. 159–178. Doya, Kenji et al. (2007). Bayesian brain: Probabilistic approaches to neural coding. MIT press. Feldman, Harriet and Karl Friston (2010). “Attention, uncertainty, and free-energy”. In: Frontiers in human neuroscience 4. Frey, Brendan J, Geoffrey E Hinton, and Peter Dayan (1996). “Does the wake-sleep algorithm produce good density estimators?” In: Advances in neural information processing systems, pp. 661–667. Friston, Karl (2005). “A theory of cortical responses”. In: Philosophical Transactions of the Royal Society of London B: Biological Sciences 360.1456, pp. 815–836. – (2008). “Hierarchical models in the brain”. In: PLoS computational biology 4.11, e1000211. 11 Friston, Karl (2009). “The free-energy principle: a rough guide to the brain?” In: Trends in cognitive sciences 13.7, pp. 293–301. Friston, Karl, Jean Daunizeau, and Stefan J Kiebel (2009). “Reinforcement learning or active inference?” In: PloS one 4.7, e6421. Friston, Karl, Thomas FitzGerald, et al. (2017). “Active inference: a process theory”. In: Neural computation 29.1, pp. 1–49. Gershman, Samuel J (2019). “What does the free energy principle tell us about the brain?” In: arXiv preprint arXiv:1901.07945. Gershman, Samuel and Noah Goodman (2014). “Amortized inference in probabilistic reasoning”. In: Proceedings of the Cognitive Science Society. Vol. 36. 36. Gillon, Colleen J et al. (2021). “Learning from unexpected events in the neocortical microcircuit”. In: bioRxiv. Gregor, Karol et al. (2014). “Deep autoregressive networks”. In: Proceedings of the International Conference on Machine Learning (ICML), pp. 1242–1250. Hinton, Geoffrey E and Drew Van Camp (1993). “Keeping the neural networks simple by minimizing the description length of the weights”. In: Proceedings of the sixth annual conference on Computational learning theory. ACM, pp. 5–13. Hopfield, John J (1982). “Neural networks and physical systems with emergent collective computational abilities”. In: Proceedings of the national academy of sciences 79.8, pp. 2554–2558. Kalman, Rudolph Emil (1960). “A new approach to linear filtering and prediction problems”. In: Journal of Basic Engineering 82.1, pp. 35–45. Kanai, Ryota et al. (2015). “Cerebral hierarchies: predictive processing, precision and the pulvinar”. In: Phil. Trans. R. Soc. B 370.1668, p. 20140169. Keller, Georg B and Thomas D Mrsic-Flogel (2018). “Predictive processing: a canonical cortical computation”. In: Neuron 100.2, pp. 424–435. Kingma, Durk P, Tim Salimans, et al. (2016). “Improved variational inference with inverse autoregressive flow”. In: Advances in neural information processing systems, pp. 4743–4751. Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Krizhevsky, Alex and Geoffrey E Hinton (2009). “Learning multiple layers of features from tiny images”. In: Laughlin, Simon (1981). “A simple coding procedure enhances a neuron’s information capacity”. In: Zeitschrift für Naturforschung c 36.9-10, pp. 910–912. LeCun, Yann, Yoshua Bengio, and Geoffrey E Hinton (2015). “Deep learning”. In: nature 521.7553, pp. 436–444. 12 Levine, Sergey (2018). “Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review”. In: arXiv preprint arXiv:1805.00909. Lotter, William, Gabriel Kreiman, and David Cox (2017). “Deep predictive coding networks for video prediction and unsupervised learning”. In: International Conference on Learning Representations. – (2018). “A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception”. In: arXiv preprint arXiv:1805.10734. MacKay, D M (1956). “The epistemological problem for automata”. In: Automata studies, pp. 235–252. McCulloch, Warren S and Walter Pitts (1943). “A logical calculus of the ideas immanent in nervous activity”. In: The bulletin of mathematical biophysics 5.4, pp. 115–133. Millidge, Beren et al. (2020). “On the Relationship Between Active Inference and Control as Inference”. In: arXiv preprint arXiv:2006.12964. Mnih, Andriy and Karol Gregor (2014). “Neural Variational Inference and Learning in Belief Networks”. In: International Conference on Machine Learning, pp. 1791– 1799. Mumford, David (1991). “On the computational architecture of the neocortex”. In: Biological cybernetics 65.2, pp. 135–145. – (1992). “On the computational architecture of the neocortex”. In: Biological cybernetics 66.3, pp. 241–251. Neal, Radford M and Geoffrey E Hinton (1998). “A view of the EM algorithm that justifies incremental, sparse, and other variants”. In: Learning in graphical models. Springer, pp. 355–368. Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu (2016). “Pixel Recurrent Neural Networks”. In: International Conference on Machine Learning, pp. 1747–1756. Parra, Lucas, Gustavo Deco, and Stefan Miesbach (1995). “Redundancy reduction with information-preserving nonlinear maps”. In: Network: Computation in Neural Systems 6.1, pp. 61–72. Pearl, Judea (1986). “Fusion, propagation, and structuring in belief networks”. In: Artificial Intelligence 29.3, pp. 241–288. Prieto, Alberto et al. (2016). “Neural networks: An overview of early research, current frameworks and new challenges”. In: Neurocomputing 214, pp. 242–268. Rao, Rajesh PN and Dana H Ballard (1999). “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” In: Nature neuroscience 2.1. 13 Rezende, Danilo Jimenez and Shakir Mohamed (2015). “Variational Inference with Normalizing Flows”. In: International Conference on Machine Learning, pp. 1530–1538. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Rosenblatt, Frank (1958). “The perceptron: a probabilistic model for information storage and organization in the brain.” In: Psychological review 65.6, p. 386. Rosenblueth, Arturo, Norbert Wiener, and Julian Bigelow (1943). “Behavior, purpose and teleology”. In: Philosophy of science 10.1, pp. 18–24. Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). “Learning representations by back-propagating errors.” In: Nature. Schmidhuber, Jürgen (2015). “Deep learning in neural networks: An overview”. In: Neural networks 61, pp. 85–117. Shannon, Claude E (1948). “A mathematical theory of communication”. In: The Bell system technical journal 27.3, pp. 379–423. Spratling, Michael W (2008). “Reconciling predictive coding and biased competition models of cortical function”. In: Frontiers in computational neuroscience 2, p. 4. Srinivasan, Mandyam Veerambudi, Simon Laughlin, and Andreas Dubs (1982). “Predictive coding: a fresh view of inhibition in the retina”. In: Proceedings of the Royal Society of London. Series B. Biological Sciences 216.1205, pp. 427–459. Todorov, Emanuel (2008). “General duality between optimal control and estimation”. In: Decision and Control, 2008. CDC 2008. 47th IEEE Conference on. IEEE, pp. 4286–4292. Todorov, Emanuel, Tom Erez, and Yuval Tassa (2012). “Mujoco: A physics engine for model-based control”. In: Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, pp. 5026–5033. Toussaint, Marc and Amos Storkey (2006). “Probabilistic inference for solving discrete and continuous state Markov Decision Processes”. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp. 945–952. Uria, Benigno, Iain Murray, and Hugo Larochelle (2014). “A deep and tractable density estimator”. In: International Conference on Machine Learning, pp. 467– 475. Walsh, Kevin S et al. (2020). “Evaluating the neurophysiological evidence for predictive processing as a model of perception”. In: Annals of the New York Academy of Sciences 1464.1, p. 242. Wiener, Norbert (1942). The Interpolation, Extrapolation and Smoothing of Stationary Time Series. NDRC Report. 14 Wiener, Norbert (1948). Cybernetics or Control and Communication in the Animal and the Machine. MIT press. 15 Chapter 2 BACKGROUND 2.1 Introduction This chapter presents the mathematical concepts underlying the technical contributions developed throughout this thesis. For clarity, the material is presented using the notation for state estimation (Part II), and we delay presenting the concepts more specifically related to action selection until Part III. 2.2 Probabilistic Models Set-Up: Probability Distributions & Maximum Likelihood Consider a random variable, 𝑋 ∈ R 𝑀 , with a corresponding distribution, 𝑝 data (𝑋), defining the probability of observing each possible observation, 𝑋 = x. We will use the shorthand notation 𝑝 data (x) to denote the probability 𝑝 data (𝑋 = x). This distribution is the result of an underlying data generation process, e.g. the emission and scattering of photons. While we do not have direct access to 𝑝 data , we can sample observations, x ∼ 𝑝 data (x), yielding an empirical distribution, 𝑝bdata (x). Often, we wish to model 𝑝 data , for instance, to predict or compress observations of 𝑋. We refer to this model as 𝑝 𝜃 (x), with parameters 𝜃. A natural approach for estimating the model involves minimizing some divergence measure between 𝑝 data (x) and 𝑝 𝜃 (x) with respect to the model parameters, 𝜃. The Kullback Leibler (KL) divergence, denoted 𝐷 KL , is a common choice of divergence measure: 𝜃 ∗ ← arg min 𝐷 KL ( 𝑝 data (x)|| 𝑝 𝜃 (x)), (2.1) 𝜃 𝜃 ∗ ← arg min Ex∼𝑝data (x) [log 𝑝 data (x) − log 𝑝 𝜃 (x)] , (2.2) 𝜃 ∗ ← arg max Ex∼𝑝data (x) [log 𝑝 𝜃 (x)] . (2.3) 𝜃 𝜃 Here, E denotes the expectation operator. We have gone from Eq. 2.1 to Eq. 2.2 using the definition of the KL divergence. Because 𝑝 data (x) does not depend on 𝜃, we can omit it from the optimization, expressing the equivalent maximization in Eq. 2.3. This is the standard maximum log-likelihood objective, which is found throughout machine learning and probabilistic modeling (Murphy, 2012; Goodfellow, Y. Bengio, and Courville, 2016). In practice, we do not have access to 𝑝 data (x) and must instead approximate the objective using data samples, i.e. using 𝑝bdata (x). With x (𝑖) as the 𝑖 th 16 <latexit sha1_base64="NZ5I9Uz3EubyGC9Sm16VZOY8I+E=">AAACdnicdVHLSgMxFE3HV61vXQoSLEVXZaYKuiy6cVnBtkIdSiZz2waTSUgyYhn6BW714/wTl2baWfShFwKHcx/n5N5IcWas73+XvLX1jc2t8nZlZ3dv/+Dw6LhjZKoptKnkUj9HxABnCbQtsxyelQYiIg7d6PU+z3ffQBsmkyc7VhAKMkzYgFFiHfWo+odVv+5PA6+CoABVVESrf1Tqv8SSpgISSzkxphf4yoYZ0ZZRDpPKS2pAEfpKhtBzMCECTJhNnU5wzTExHkjtXmLxlJ3vyIgwZiwiVymIHZnlXE7+leuldnAbZixRqYWEzoQGKcdW4vzbOGYaqOVjBwjVzHnFdEQ0odYtZ2HSUxBmubl8zIJ8JCaVGp5nchvKivfFOsKH0imMxD80o6sNykDqtipjt0B3kmD5AKug06gHV/XG43W1eVccp4xO0Tm6RAG6QU30gFqojSgC9IE+0Vfpxzvzat7FrNQrFT0naCE8/xcndML9</latexit> p pdata (x) <latexit sha1_base64="KMQ8ygvEEBrHwGNZcCadnKEw6ng=">AAACk3icdVHLahsxFJWnj6TuI05LV+1C1ATSjZlJCyl0E9IuuimkECcBexjuaO7YItJISHeKzTCbfk237d/0b6pxvIjj9oLgcO7r6NzcKukpjv/0onv3Hzzc2X3Uf/zk6bO9wf7zC29qJ3AsjDLuKgePSlY4JkkKr6xD0LnCy/z6U5e//I7OS1Od09JiqmFWyVIKoEBlg9c2mxIuyOmmAIKWH0410Dwvm0X7NhsM41G8Cr4NkjUYsnWcZfu9bFoYUWusSCjwfpLEltIGHEmhsO1Pa48WxDXMcBJgBRp92qy+0fKDwBS8NC68iviKvd3RgPZ+qfNQ2Wn0d3Md+a/cpKbyQ9rIytaElbhZVNaKk+GdJ7yQDgWpZQAgnAxauZiDA0HBuY1J50nadOK6MRvrc932D/htppNhSS8260DNTNgw1/+hpdhusB7r4KopgoHhJMndA2yDi6NR8m509O398OR0fZxd9oq9YYcsYcfshH1hZ2zMBPvBfrJf7Hf0MvoYnUafb0qj3rrnBduI6OtfGl3N3Q==</latexit> <latexit sha1_base64="Min9X1kSrlMsgf0t/tlM/bkvIlw=">AAACo3icdVFNa9tAEF2rX4nTNk57zGWpMSRQjJQE2mNoL4Vc0mAnAUuI1WpkL9nVLrujYiP0J/prek3+Rf9NVo4hcdwOLDzevJl5O5MZKRyG4d9O8OLlq9dvtra7O2/fvd/t7X24dLqyHMZcS22vM+ZAihLGKFDCtbHAVCbhKrv53uavfoF1QpcjXBhIFJuWohCcoafS3udYMZxlRT1vaOyEoiaNEeZoVZ0zZA09eBQcpr1+OAyXQTdBtAJ9sorzdK+TxrnmlYISuWTOTaLQYFIzi4JLaLpx5cAwfsOmMPGwZApcUi+/1dCBZ3JaaOtfiXTJPq2omXJuoTKvbD2657mW/FduUmHxNalFaSqEkj8MKipJUdN2RzQXFjjKhQeMW+G9Uj5jlnH0m1zrNIqSujXXtlkbn6mmO6BPmdaGQTVf1zE51X7CTP2HFnyzwDio/FZ17hfoTxI9P8AmuDwaRsfDo58n/dNvq+NskX3yiRyQiHwhp+QHOSdjwslv8ofckrtgEJwFF8HoQRp0VjUfyVoEyT0JRtSg</latexit> <latexit sha1_base64="DJHvJodbDQ1zy9j80H0t+wyUyzg=">AAACjHicdVFdSxtBFJ1sW03TqtHSJ1+GBsG+hF0VFEohtCA+KiQqJMsyO7mbDM7sDDN3JWHJj/G1/UX+G2djHvKhFy4czv06nJsaKRyG4XMt+PDx09Z2/XPjy9ed3b3m/sGt04Xl0ONaanufMgdS5NBDgRLujQWmUgl36cPfqn73CNYJnXdxaiBWbJSLTHCGnkqa300ywDEgo8cDxXCcZuVk9jNptsJ2OA+6CaIFaJFFXCf7tWQw1LxQkCOXzLl+FBqMS2ZRcAmzxqBwYBh/YCPoe5gzBS4u5/pn9MgzQ5pp6zNHOmeXJ0qmnJuq1HdWGt16rSLfqvULzC7iUuSmQMj566GskBQ1rcygQ2GBo5x6wLgVXivlY2YZR2/ZyqZuFJeVuGrNyvlUzRpHdJmpZBhUk9U+JkfaXxird2jBNweMg8K7qofeQP+SaP0Bm+D2pB2dtk9uzlqdP4vn1Mkh+UGOSUTOSYdckWvSI5yU5In8I/+D3eAs+BX8fm0NaouZb2QlgssXUBDKdw==</latexit> x ⇠ pdata (x) p✓ (x) X <latexit sha1_base64="ZTIWoTksoLoiyDHEgNaAIVk1a3s=">AAACdnicdVHLTgIxFC3jC/EFujQxjYToisygiS6JblxCwiuBCemUCzS200nbMZIJX+BWP84/cWkHWDCgN2lycu7jnN4bRJxp47rfOWdnd2//IH9YODo+OT0rls47WsaKQptKLlUvIBo4C6FtmOHQixQQEXDoBq/Pab77BkozGbbMLAJfkEnIxowSY6lmb1gsu1V3EXgbeCtQRqtoDEu54WAkaSwgNJQTrfueGxk/IcowymFeGMQaIkJfyQT6FoZEgPaThdM5rlhmhMdS2RcavGDXOxIitJ6JwFYKYqZ6M5eSf+X6sRk/+gkLo9hASJdC45hjI3H6bTxiCqjhMwsIVcx6xXRKFKHGLiczqeX5SWouHZORD8S8UMHrTGojMuI9W0f4RFqFqfiHZnS7IdIQ263KkV2gPYm3eYBt0KlVvbtqrXlfrj+tjpNHl+ga3SIPPaA6ekEN1EYUAfpAn+gr9+NcORXnZlnq5FY9FygTjvsL9bXC5Q==</latexit> Figure 2.1: Maximum Likelihood Estimation. A data distribution, 𝑝 data , produces samples, x, of a random variable 𝑋. Maximum likelihood estimation attempts to fit a model distribution, 𝑝 𝜃 , to the empirical distribution of samples by maximizing the log-likelihood of data samples under the model (Eq. 2.3). observation sample out of 𝑁 total samples, we can approximate the objective as the following empirical average: 𝑁 1 Õ Ex∼𝑝data (x) [log 𝑝 𝜃 (x)] ≈ Ex∼b log 𝑝 𝜃 (x (𝑖) ). 𝑝 data (x) [log 𝑝 𝜃 (x)] = 𝑁 𝑖=1 (2.4) In summary, we can frame probabilistic modeling as the process of maximizing the log-likelihood, log 𝑝 𝜃 (x), of data observations under our model. A diagram of this setup is shown in Figure 2.1. Model Formulation Formulating a probabilistic model involves considering the dependency structure of the model and the parameterization of these dependencies. As with all models, there is a bias-variance decomposition of modeling error. Without considering a sufficiently flexible dependency structure and parameterization, the model will be inherently limited, i.e. biased, in its capacity to accurately model the data. Yet, with an overly expressive model, containing many dependencies and parameters, there is a risk of overfitting to the training examples, the outcome of a model with high variance. 17 Dependency Structure The dependency structure of a probabilistic model is the set of conditional dependencies between variables. One common form of dependency structure is that of autoregression (Frey, G. E. Hinton, and Dayan, 1996; Y. Bengio and S. Bengio, 2000), which utilizes the chain rule of probability to model dependencies between variables: 𝑀 Ö 𝑝 𝜃 (x) = 𝑝 𝜃 (𝑥 𝑗 |𝑥 < 𝑗 ). (2.5) 𝑗=1 Here, we have induced an arbitrary ordering over the 𝑀 dimensions of x, allowing us to factor the joint distribution over dimensions, 𝑝 𝜃 (x), into a product of 𝑀 conditional distributions, each conditioned on the previous dimensions, 𝑥 < 𝑗 . Slightly abusing notation, at 𝑗 = 1, we have 𝑝 𝜃 (𝑥 𝑗 |𝑥 < 𝑗 ) = 𝑝 𝜃 (𝑥1 ). A natural use-case for this dependency structure arises in modeling sequential data, where time provides an ordering over a sequence of 𝑇 observed variables, x1:𝑇 : 𝑝 𝜃 (x1:𝑇 ) = 𝑇 Ö 𝑝 𝜃 (x𝑡 |x<𝑡 ). (2.6) 𝑡=1 Although such models are conventionally formulated in forward temporal order, this is truly a modeling assumption. Likewise, while the chain rule of probability dictates that we must consider all previous variables, there may be cases where it is safe to assume conditional independence outside of some window. In the extreme case, in which we only consider pairwise dependencies, we arrive at a Markov chain: Î 𝑝 𝜃 (x1:𝑇 ) = 𝑇𝑡=1 𝑝 𝜃 (x𝑡 |x𝑡−1 ). Autoregressive models are also referred to as “fully-visible” models (Frey, G. E. Hinton, and Dayan, 1996), as dependencies are only explicitly modeled between observed variables. However, we can also model such dependencies by introducing latent variables, denoted as 𝑍. Formally, a latent variable model is defined by the joint distribution 𝑝 𝜃 (x, z) = 𝑝 𝜃 (x|z) 𝑝 𝜃 (z), (2.7) where 𝑝 𝜃 (x|z) is the conditional likelihood and 𝑝 𝜃 (z) is the prior. Again, we have used the shorthand notation 𝑝 𝜃 (x, z) to denote 𝑝 𝜃 (𝑋 = x, 𝑍 = z). Introducing latent variables is one of, if not, the primary technique for increasing the flexibility of a probabilistic model. This is because evaluating the probability of an observation now requires marginalizing over the latent variables. If 𝑍 is a continuous variable, ∫ this involves integration, 𝑝 𝜃 (x) = 𝑝 𝜃 (x, z)𝑑z, and if 𝑍 is discrete, this involves 18 Figure 2.2: Dependency Structures. Circles denote random variables, with gray denoting observed variables and white denoting latent variables. Arrows denote probabilistic conditional dependencies. From left to right: autoregressive model (Eqs. 2.5 & 2.6), latent variable model (Eq. 2.7), hierarchical latent variable model (Eq. 2.10), autoregressive or sequential latent variable model (Eq. 2.11). For clarity, 34 we have drawn a subset of the possible dependencies in the final model. summation, 𝑝 𝜃 (x) = Í z 𝑝 𝜃 (x, z). In either case, we have 𝑝 𝜃 (x) = Ez∼𝑝 𝜃 (z) [ 𝑝 𝜃 (x|z)] , (2.8) which illustrates that 𝑝 𝜃 (x) is a mixture distribution, with each mixture component, 𝑝 𝜃 (x|z), weighted according to 𝑝 𝜃 (z). Thus, even when restricting 𝑝 𝜃 (x|z) to simple distribution forms, such as Gaussian distributions, 𝑝 𝜃 (x) can take on flexible forms that do not have closed form analytical expressions. In this way, 𝑍 can implicitly model dependencies in 𝑋, assigning higher probability to particular regions of the observation space. However, increasing flexibility through latent variables comes with increasing computational overhead. In general, marginalizing over 𝑍 is not analytically tractable, particularly with continuous latent variables or complex conditional likelihoods. This requires us to either 1) adopt approximation techniques, which we discuss in Section 2.3, or 2) restrict the form of the model to ensure computationally tractable evaluation of 𝑝 𝜃 (x). This latter approach is the basis of flow-based models (Tabak and Turner, 2013; Rippel and Adams, 2013; Dinh, Krueger, and Y. Bengio, 2015), which define the conditional dependency between 𝑋 and 𝑍 in terms of an invertible transform, x = 𝑓𝜃 (z) and z = 𝑓𝜃−1 (x). We can then express 𝑝 𝜃 (x) using the change of variables formula:   −1 𝜕x 𝑝 𝜃 (x) = 𝑝 𝜃 (z) det , (2.9) 𝜕z 19 where 𝜕x Jacobian of the transform and det(·) denotes matrix determinant. 𝜕z is the   −1 The term det 𝜕x can be interpreted as the local scaling of space when moving 𝜕z from 𝑍 to 𝑋, conserving probability mass in the transform. Flow-based models, also referred to as normalizing flows (Rezende and Mohamed, 2015), are the basis of the classical technique of independent components analysis (ICA) (Bell and Sejnowski, 1997; Hyvärinen and Oja, 2000) and non-linear generalizations (S. S. Chen and Gopinath, 2001; Laparra, Camps-Valls, and Malo, 2011). As such, these models can serve as a general-purpose mechanism for adding and removing statistical dependencies between variables. Although flow-based models avoid the intractability of marginalization, their requirement of invertibility may be overly restrictive or undesirable in some contexts (Cornish et al., 2020). And while the change of variables formula can also be applied to non-invertible transforms (Cvitkovic and Koliander, 2019), it raises computational intractabilities. This motivates the use of approximate techniques for training latent variables models (Section 2.3). While we have presented autoregression and latent variables separately, these techniques can, in fact, be combined in numerous ways to model dependencies. For instance, one can create hierarchical latent variable models (Dayan et al., 1995), incorporating autoregressive dependencies between latent variables. Considering 𝐿   levels of latent variables, 𝑍 1:𝐿 = 𝑍 1 , . . . , 𝑍 𝐿 , we can express the joint distribution as 𝐿 Ö 1:𝐿 1:𝐿 𝑝 𝜃 (x, z ) = 𝑝 𝜃 (x|z ) 𝑝 𝜃 (zℓ |zℓ+1:𝐿 ). (2.10) ℓ=1 From this perspective, hierarchical latent variable models are a repeated application of the latent variables technique in order to create increasingly complex empirical priors (Efron and Morris, 1973). We can also consider incorporating latent variables within sequential (autoregressive) probabilistic models, giving rise to sequential latent variable models. Considering a single level of latent variables in a corresponding sequence, 𝑍1:𝑇 , we generally have the following joint distribution: 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) = 𝑇 Ö 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ), (2.11) 𝑡=1 where we have again assumed a forward sequential ordering. By restricting the dependency structure and distribution forms in sequential latent variable models, we recover familiar special cases, such as hidden Markov models or linear Gaussian state-space models (Murphy, 2012). Beyond hierarchical and sequential latent variable models, there are a variety of other ways to combine autoregression and 20 latent variables (Gulrajani et al., 2017; Razavi, Aaron van den Oord, and Vinyals, 2019). The remaining chapters focus on hierarchical (Eq. 2.10) and sequential (Eq. 2.11) latent variable models, though models with more flexible hierarchical, sequential, and spatial dependencies will be required to advance the frontier of probabilistic modeling. Parameterizing the Model In the previous section, we discussed the dependency structure of probabilistic models. The probability distributions that define these dependencies are ultimately functions. In this section, we discuss possible forms that these functions may take. We restrict our focus here to parametric distributions, which are defined by one or more distribution parameters. The canonical example is the Gaussian (or Normal) distribution, N (𝑥; 𝜇, 𝜎 2 ), which is defined by a mean, 𝜇, and variance, 𝜎 2 . This can be extended to the multivariate setting, where x ∈ R 𝑀 is modeled with a mean vector, µ, and covariance matrix, 𝚺, with the probability density written as   1 −1 | −1 (x − µ) 𝚺 (x − µ) . (2.12) N (x; µ, 𝚺) = exp 2 (2𝜋) 𝑀/2 det(𝚺) 1/2 For convenience, we may also consider diagonal covariance matrices, 𝚺 = diag(σ 2 ), simplifying the parameterization and resulting calculations. In particular, the special case where 𝚺 = I 𝑀 , the 𝑀 × 𝑀 identity matrix, the log-density, up to a constant, becomes the familiar mean squared error, 1 log N (x; µ, I) = − ||x − µ||22 + const. 2 (2.13) With a parametric distribution, conditional dependencies are mediated by the distribution parameters, which are functions of the conditioning variables. For example, we can express an autoregressive Gaussian distribution (of the form in Eq. 2.5) through conditional densities, 𝑝 𝜃 (𝑥 𝑗 |𝑥 < 𝑗 ) = N (𝑥 𝑗 ; 𝜇𝜃 (𝑥 < 𝑗 ), 𝜎𝜃2 (𝑥 < 𝑗 )), where 𝜇𝜃 and 𝜎𝜃2 are functions taking 𝑥 < 𝑗 as input. A similar form applies to autoregressive models on sequences of vector inputs (Eq. 2.6), with 𝑝 𝜃 (x𝑡 |x<𝑡 ) = N (x𝑡 ; µ𝜃 (x<𝑡 ), 𝚺 𝜃 (x<𝑡 )). Likewise, in a latent variable model (Eq. 2.7), we can express a Gaussian conditional likelihood as 𝑝 𝜃 (x|z) = N (x; µ𝜃 (z), 𝚺 𝜃 (z)). Note that in the above examples, we have overloaded notation, simply using a subscript 𝜃 for all functions. In practice, while it is not uncommon to share parameters across functions, this is ultimately a modeling choice. 21 The functions supplying each of the distribution parameters can range in complexity, from constant to highly non-linear. Classical modeling techniques often employ linear functions. For instance, in a latent variable model, we could parameterize the mean as a linear function of z: µ𝜃 (z) = Wz + b, (2.14) where W is a matrix of weights and b is a bias vector. Models of this form underlie factor analysis, probabilistic principal components analysis (Tipping and Bishop, 1999), independent components analysis (Bell and Sejnowski, 1997; Hyvärinen and Oja, 2000), and sparse coding (Olshausen and Field, 1996). Linear autoregressive models are also the basis of many classical time-series models and are common across fields that use statistical methods. While linear models are relatively computationally efficient, they are often too limited to accurately model complex data distributions, e.g. those found in natural images or audio. Recent improvements in deep learning (Goodfellow, Y. Bengio, and Courville, 2016) have provided probabilistic models with a more expressive class of non-linear functions, improving their modeling capacity. In these models, the distribution parameters are parameterized with deep networks, which are then trained by backpropagating (Rumelhart, G. E. Hinton, and Williams, 1986) the gradient of the log-likelihood objective, ∇𝜃 Ex∼b 𝑝 data [log 𝑝 𝜃 (x)], back through the layers of the network. Typically, this is estimated using a mini-batch of data examples, rather than the full empirical distribution as in Eq. 2.4. Deep autoregressive models and deep latent variable models have enabled recent advances across an array of areas, including speech (Graves, 2013; Aäron van den Oord et al., 2016), natural language (Sutskever, Vinyals, and Le, 2014; Radford et al., 2019), images (Razavi, Aaron van den Oord, and Vinyals, 2019), video (Kumar et al., 2020), reinforcement learning (Chua et al., 2018; Ha and Schmidhuber, 2018) and many others. We visualize a simplified probabilistic computation graph for a deep autoregressive model in Figure 2.3. This diagram translates the autoregressive dependency structure from Figure 2.2, which only depicts the variables and their dependencies in the model, into a more detailed visualization, breaking the variables into their corresponding distributions and terms in the log-likelihood objective. Here, green circles denote the conditional likelihood at each step, containing a Gaussian mean and standard deviation, which are parameterized by a deep network. The log-likelihood, log 𝑝 𝜃 (x𝑡 |x<𝑡 ), evaluated at the data observation, x𝑡 ∼ 𝑝 data (x𝑡 |x<𝑡 ) (gray circle), provides the 22 µ✓ <latexit sha1_base64="l0Ke2CNbY3w+39J9mtQy0ZuO9Ck=">AAAChnicdVHLSgMxFE3HV62vqks3wSK4KjM+0GXRjUsFq0JnGDLpbRtMJkNyI5ahX+JWP8q/MVO7sFYvXDic+zqcmxVSWAzDz1qwtLyyulZfb2xsbm3vNHf3Hqx2hkOXa6nNU8YsSJFDFwVKeCoMMJVJeMyer6v64wsYK3R+j+MCEsWGuRgIztBTaXMnzlQZKzdJYxwBsrTZCtvhNOgiiGagRWZxm+7W0rivuVOQI5fM2l4UFpiUzKDgEiaN2FkoGH9mQ+h5mDMFNimnyif0yDN9OtDGZ450yv6cKJmydqwy36kYjuzvWkX+Ves5HFwmpcgLh5Dz70MDJylqWtlA+8IARzn2gHEjvFbKR8wwjt6suU33UVJW4qo1c+czNWkc0Z9MJaNA9Trfx+RQ+wsj9Q8t+OJAYcF5V3XfG+hfEv1+wCJ4OGlHp+2Tu7NW52r2nDo5IIfkmETkgnTIDbklXcKJI2/knXwE9aAdnAcX361BbTazT+Yi6HwBrh7IfQ==</latexit> <latexit sha1_base64="hsjE9FCjkzXNO2FF+fHIUg7mGLw=">AAACoHicdVHBThsxEHUWSmmgNLTHXiwiJLhEu1AJDhwQveTWIBFAJKuV15lNLOy1Zc9WRMt+A1/DFb6jf1NvyCEh7UiWnt/MeJ7fpEYKh2H4pxGsrX/Y+Lj5qbm1/XnnS2v367XTheXQ51pqe5syB1Lk0EeBEm6NBaZSCTfp/c86f/MbrBM6v8KpgVixcS4ywRl6KmkdmmSIE0BGD4aK4STNyocqQfpIF67lGVaHSasddsJZ0FUQzUGbzKOX7DaS4UjzQkGOXDLnBlFoMC6ZRcElVM1h4cAwfs/GMPAwZwpcXM7+VNF9z4xopq0/OdIZu9hRMuXcVKW+shbq3udq8l+5QYHZaVyK3BQIOX8blBWSoqa1QXQkLHCUUw8Yt8JrpXzCLOPobVx66SqKy1pc/czS+FRVzX26yNQyDKqH5Tomx9pPmKj/0IKvNhgHhXdVj7yBfiXR+wWsguujTnTcObr80T6/mC9nk3wne+SAROSEnJMu6ZE+4eSJPJMX8hrsBd3gV3D5Vho05j3fyFIEd38BY6vTEw==</latexit> p✓ (xt |x<t ) log p✓ (xt |x<t ) <latexit sha1_base64="L82yc0PcdKQbIVR4FVe9NQvnOgE=">AAACpXicdVHBThsxEHWWtkBa2kCPvViNUOES7UIlOHBA9NJLJZCSgJRdrbzObGJhry17FhEt+xd8DVf6E/2bekMOCWlHsvT8Zsbz/CYzUjgMwz+tYOPN23ebW9vt9x92Pn7q7O4NnS4thwHXUtubjDmQooABCpRwYywwlUm4zm5/NPnrO7BO6KKPMwOJYpNC5IIz9FTa6cVST6hJY5wCMnoQK4bTLK/u6xTpA126VmdYH6adbtgL50HXQbQAXbKIy3S3lcZjzUsFBXLJnBtFocGkYhYFl1C349KBYfyWTWDkYcEUuKSaf6ym+54Z01xbfwqkc3a5o2LKuZnKfGUj1L3ONeS/cqMS89OkEoUpEQr+MigvJUVNG5foWFjgKGceMG6F10r5lFnG0Xu58lI/SqpGXPPMyvhM1e19usw0Mgyq+9U6JifaT5iq/9CCrzcYB6V3VY+9gX4l0esFrIPhUS867h1dfe+eXyyWs0W+kK/kgETkhJyTn+SSDAgnj+SJPJPfwbfgV9APhi+lQWvR85msRJD+BQgy1QM=</latexit> <latexit sha1_base64="dWNC+c6RW6+WgIuGD7J2Bwe3Emg=">AAACiXicdVFNSyNBEO3M+pGNX1GPXhqD4CnMqKB4EnPZowtGhcww1HQqSWP39NBdI4Yhf8Wr+5f232xPzMGYtaDg8err8SorlHQUhn8bwY+19Y3N5s/W1vbO7l57/+DBmdIK7AujjH3KwKGSOfZJksKnwiLoTOFj9tyr648vaJ00+T1NC0w0jHM5kgLIU2n7IM50FTs51jBLY5ogQdruhN1wHnwVRAvQYYu4S/cbaTw0otSYk1Dg3CAKC0oqsCSFwlkrLh0WIJ5hjAMPc9DokmoufsZPPDPkI2N95sTn7OeJCrRzU535Tg00cV9rNfm/2qCk0VVSybwoCXPxcWhUKk6G107wobQoSE09AGGl18rFBCwI8n4tbbqPkqoWV69ZOp/pWeuEf2ZqGQXp1+U+UGPjL0z0N7QUqwOFw9K7aobeQP+S6OsDVsHDWTc67579vujc3C6e02RH7Jidsohdshv2i92xPhPslb2xd/Yn2Aqi4Cq4/mgNGouZQ7YUQe8fpfnJyg==</latexit> ✓ xt ⇠ pdata <latexit sha1_base64="W0flu+bhmcsHUKZ1gjE/g4gfuv4=">AAACmHicdVHLjtMwFHXDayivDuxgY6hGYlUlAxIsK1gAuwFNZ0ZqoujGuWmtsWPLvkGtoqz5GrbwLfwNTqeL6RSuZOno3NfxuYVV0lMc/xlEt27fuXvv4P7wwcNHj5+MDp+eedM4gTNhlHEXBXhUssYZSVJ4YR2CLhSeF5cf+/z5d3RemvqU1hYzDYtaVlIABSofvUw10LKo2lWXE0+91NzmKeGKnG5LIOjy0TiexJvg+yDZgjHbxkl+OMjT0ohGY01CgffzJLaUteBICoXdMG08WhCXsMB5gDVo9Fm7+UvHjwJT8sq48GriG/Z6Rwva+7UuQmWv3N/M9eS/cvOGqvdZK2vbENbialHVKE6G98bwUjoUpNYBgHAyaOViCQ4EBft2Jp0mWduL68fsrC90Nzzi15lehiW92q0DtTBhw1L/h5Ziv8F6bIKrpgwGhpMkNw+wD86OJ8mbyfHXt+Pph+1xDtgL9oq9Zgl7x6bsMzthMybYD/aT/WK/o+fRNPoUfbkqjQbbnmdsJ6JvfwHIjdBW</latexit> Figure 2.3: Model Parameterization & Computation Graph. The diagram depicts 11 autoregressive Gaussian model. Green a simplified computation graph for a deep circles denote the conditional likelihood distribution at each step, while gray circles again denote the (distribution of) data observations. Smaller red circles denote each of the log-likelihood terms in the objective. Gradients w.r.t. these terms are backpropagated through the networks parameterizing the model’s distribution parameters (red dotted lines). objective (red dot). The gradient of this objective w.r.t. the network parameters is calculated through backpropagation (red dotted line). This general depiction of dependencies, distributions, and log-probabilities (or differences of log-probabilities) is used throughout this thesis to describe the various computational setups. Purely autoregressive models (without latent variables) have proven useful in many domains, often obtaining better log-likelihoods as compared with latent variable models. However, there are a number of reasons to prefer latent variable models in some contexts. First, autoregressive sampling is inherently sequential, and this linear computational scaling becomes costly in high-dimensional domains. Second, latent variables provide a representational space for downstream tasks, compression, and overall data analysis. Finally, latent variables provide added flexibility, which is particularly useful for modeling continuous random variables with relatively simple, e.g. Gaussian, conditional distributions. For these reasons, we require methods for handling the latent marginalization in Eq. 2.8. Variational inference is one such method. 23 2.3 Variational Inference Derivation As we saw in the previous section, training latent variable models through maximum likelihood requires evaluating log 𝑝 𝜃 (x). However, particularly with continuous latent ∫ variables, evaluating 𝑝 𝜃 (x) = 𝑝 𝜃 (x, z)𝑑z is generally computationally intractable. This problem is only exacerbated in deep latent variable models, where computing 𝑝 𝜃 (z) and 𝑝 𝜃 (x|z) is more expensive. Thus, we require some technique for tractably estimating log 𝑝 𝜃 (x) without resorting to exact marginalization. Variational inference (G. E. Hinton and Van Camp, 1993; Jordan et al., 1998) approaches this problem by introducing an approximate posterior distribution, 𝑞(z|x), which provides a tractable lower bound, L (x; 𝑞, 𝜃) ≤ log 𝑝 𝜃 (x), on the log-likelihood. This lower bound is variously referred to as the evidence lower bound (ELBO), variational lower bound, and the negative free energy. By tightening and maximizing the ELBO w.r.t. the model parameters, 𝜃, we can approximate maximum likelihood training while avoiding marginalization. We can interpret variational inference as converting probabilistic inference into an optimization problem. Given a family of distributions, Q, e.g., Gaussian, nonparametric, etc., variational inference attempts to find the distribution, 𝑞 ∈ Q, that minimizes 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z|x)): 𝑞(z|x) ← arg min 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z|x)), (2.15) 𝑞 where 𝑝 𝜃 (z|x) is the posterior distribution, 𝑝 𝜃 (z|x) = 𝑝 𝜃 (x, z) . 𝑝 𝜃 (x) (2.16) Because 𝑝 𝜃 (z|x) includes the intractable 𝑝 𝜃 (x), we cannot minimize the KL divergence in Eq. 2.15 directly. Instead, we can rewrite this as 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z|x)) = Ez∼𝑞(z|x) [log 𝑞(z|x) − log 𝑝 𝜃 (z|x)]    𝑝 𝜃 (x, z) = Ez∼𝑞(z|x) log 𝑞(z|x) − log 𝑝 𝜃 (x) (2.17) (2.18) = Ez∼𝑞(z|x) [log 𝑞(z|x) − log 𝑝 𝜃 (x, z)] + log 𝑝 𝜃 (x) (2.19) = −L (x; 𝑞, 𝜃) + log 𝑝 𝜃 (x). (2.20) In Eq. 2.20, we have defined L (x; 𝑞, 𝜃), as L (x; 𝑞, 𝜃) ≡ Ez∼𝑞(z|x) [log 𝑝 𝜃 (x, z) − log 𝑞(z|x)] = Ez∼𝑞(z|x) [log 𝑝 𝜃 (x|z)] − 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z)). (2.21) (2.22) 24 Algorithm 1 Variational Expectation Maximization (EM) 1: Input: model 𝑝 𝜃 (x, z), data examples x (1) , . . . , x (𝑁) ∼ 𝑝 data (x) 2: while 𝜃 not converged do 3: for x = x (1) , . . . , x (𝑁) do 4: 𝑞(z|x) ← arg max𝑞 L (x; 𝑞, 𝜃) ⊲ inference (E-step) 5: end for Í𝑁 L (x (𝑖) ; 𝑞, 𝜃) ⊲ learning (M-step) 6: 𝜃 ← arg max𝜃 𝑁1 𝑖=1 7: end while Rearranging terms in Eq. 2.20, we have log 𝑝 𝜃 (x) = L (x; 𝑞, 𝜃) + 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z|x)). (2.23) Because KL divergence is non-negative, we see that L (x; 𝑞, 𝜃) ≤ log 𝑝 𝜃 (x), with equality when 𝑞(z|x) = 𝑝 𝜃 (z|x). As the LHS of Eq. 2.23 does not depend on 𝑞(z|x), maximizing L (x; 𝑞, 𝜃) w.r.t. 𝑞 implicitly minimizes 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z|x)) w.r.t. 𝑞. Together, these statements imply that maximizing L (x; 𝑞, 𝜃) w.r.t. 𝑞 tightens the lower bound on log 𝑝 𝜃 (x). With this tightened lower bound, we can then maximize L (x; 𝑞, 𝜃) w.r.t. 𝜃. This alternating optimization process is referred to as the variational expectation maximization (EM) algorithm (Dempster, Laird, and Rubin, 1977; Neal and G. E. Hinton, 1998) (Algorithm 1), consisting of approximate inference (E-step) and learning (M-step). While Algorithm 1 iterates over the entire data set between parameter updates, one can instead perform inference for a mini-batch of data examples and perform stochastic gradient ascent on 𝜃 (Hoffman et al., 2013). This version of the algorithm, referred to as stochastic variational inference (SVI), is used in practice to train deep latent variable models. Interpreting the ELBO The primary motivation for variational inference is in providing a tractable lower bound for model training. In the process, however, we are left with an approximate posterior distribution, 𝑞(z|x). To gain insight into this distribution and the approximate inference procedure, we can rewrite the ELBO as L (x; 𝑞, 𝜃) = Ez∼𝑞(z|x) [log 𝑝 𝜃 (x|z) + log 𝑝 𝜃 (z) − log 𝑞(z|x)] . (2.24) Each of the terms in the objective guides the optimization of 𝑞(z|x). The first term quantifies the agreement with the data, i.e., reconstruction: sampling z from 𝑞, we want to assign high log-probability to the observation, x. The second term quantifies the agreement with the prior prediction: sampling z from 𝑞, we want to have high S TSRTURC X IXMI M ATAT E EP O S TS E RR SS UTCUTRUERD E DA PAPPRPO RO PO TR E IRO IO 25 structured models structured models latent space p✓ (z) <latexit sha1_base64="IEFcZ6TdJDNiYnM2MXVVhnD1rxA=">AAACjHicdVHLSiNBFK20j3HiKzrMyk1hEHQTulUYQQRxQFwqGBWSpqmu3E6KVHUVVbfF2ORj3OoXzd9MdczCGL1w4XDu63BuaqRwGIb/asHC4tLyj5Wf9dW19Y3Nxtb2ndOF5dDmWmr7kDIHUuTQRoESHowFplIJ9+nwb1W/fwTrhM5vcWQgVqyfi0xwhp5KGr9N0sUBIKP7XcVwkGbl8/ggaTTDVjgJOg+iKWiSaVwnW7Wk29O8UJAjl8y5ThQajEtmUXAJ43q3cGAYH7I+dDzMmQIXlxP9Y7rnmR7NtPWZI52wHydKppwbqdR3Vhrd51pFflXrFJidxKXITYGQ8/dDWSEpalqZQXvCAkc58oBxK7xWygfMMo7esplNt1FcVuKqNTPnUzWu79GPTCXDoHqa7WOyr/2FgfqGFnx+wDgovKu65w30L4k+P2Ae3B22oqPW4c1x8/xi+pwVskN2yT6JyB9yTq7INWkTTkryQl7JW7ARHAenwdl7a1CbzvwiMxFc/gdUOMp5</latexit>  q(z|x) Eq log p✓ (z) <latexit sha1_base64="bSdQSp0iZL3ytr3Aon9aL1YVVZc=">AAAC0nicdVHLatwwFNW4r3T6mqTLbkSHQLLoYCeFdhkaAl2mMJMEbGNkjWyLSJYiXZdMVC9Ktv2pfka/oNv2DyrPTGEm014QHJ177oNzcy24hTD80Qvu3X/w8NHW4/6Tp8+evxhs75xZ1RjKJlQJZS5yYpngNZsAB8EutGFE5oKd55fHXf78MzOWq3oMM81SScqaF5wS8FQ2iN8kkkCV5+6kza5wIlgBcSJUiZPCEOqu8N5CULibFn/Bfz/X7X7rdJZAxYCsavZbnBheVpDibDAMR+E88CaIlmCIlnGabfeyZKpoI1kNVBBr4yjUkDpigFPB2n7SWKYJvSQliz2siWQ2dXMXWrzrmSkulPGvBjxnVysckdbOZO6V3bb2bq4j/5WLGyjep47XugFW08WgohEYFO4sxVNuGAUx84BQw/2umFbEuwfe+LVO4yh13XJdm7XxuWz7u3iV6dbQIK/XdUSUyk+o5H9oTjcLtGWNd1VNvYH+JNHdA2yCs4NRdDg6+PR2ePRheZwt9Aq9RnsoQu/QEfqITtEEUfQd/US/0O9gHNwEX4PbhTToLWteorUIvv0BDFnnlA==</latexit> q(z|x) <latexit sha1_base64="GKYv3YBzSW+vcCnXm4SsDmmxo+c=">AAACknicdVHLTgIxFC3jG1+g7tg0EhLckBk00bhC3bhwoQkgCUxIp3SgoZ2ObceA4yz8Grf6Of6NHcSEh96kyem59/aenuuFjCpt218Za2V1bX1jcyu7vbO7t5/LHzSViCQmDSyYkC0PKcJoQBqaakZaoSSIe4w8esObNP/4TKSiIqjrcUhcjvoB9SlG2lDdXOEJljsc6YHnxy8JfIW/l1Fy0s0V7Yo9CbgMnCkogmncd/OZbqcncMRJoDFDSrUdO9RujKSmmJEk24kUCREeoj5pGxggTpQbT36RwJJhetAX0pxAwwk72xEjrtSYe6Yy1agWcyn5V64daf/CjWkQRpoE+GeQHzGoBUwtgT0qCdZsbADCkhqtEA+QRFgb4+ZeqjtunIpLn5kb7/EkW4KzTCoj1Hw0X4dYX5gJA/4PTfFyQ6hIZFwVPWOgWYmzuIBl0KxWnNNK9eGsWLueLmcTFMAxKAMHnIMauAX3oAEweAPv4AN8WkfWpXVl3fyUWplpzyGYC+vuG+HlzN8=</latexit> <latexit sha1_base64="60DGexcpR/Fv1prZWkg4K8IbvkM=">AAACmnicdVFNTxsxEHW2LdC0fLXH9mARIdFLtBuQ4BjBhaoXkBJASlaR15lNLOy1Zc+ihGV/AL+GK/0p/Tf1hiARAiNZen7zxjN+kxgpHIbhv1rw4eOnldW1z/UvX9c3Nre2v104nVsOXa6ltlcJcyBFBl0UKOHKWGAqkXCZXJ9U+csbsE7orINTA7Fio0ykgjP01GCrQc2gj2NARvf6iuE4SYtJSe/o8+W2/OVVYTOcBV0G0Rw0yDzOBtu1QX+oea4gQy6Zc70oNBgXzKLgEsp6P3dgGL9mI+h5mDEFLi5mvynprmeGNNXWnwzpjH1ZUTDl3FQlXlnN6F7nKvKtXC/H9CguRGZyhIw/NUpzSVHTyho6FBY4yqkHjFvhZ6V8zCzj6A1ceKkTxUU1XPXMQvtElfVd+pKpxjCoJos6Jkfadxird2jBlwuMg9y7qofeQL+S6PUClsFFqxntN1vnB4328Xw5a+QH2SF7JCKHpE1OyRnpEk7uyQN5JH+Dn8Fx8Dv48yQNavOa72Qhgs5/fj/QHw==</latexit> p✓ (x|z) <latexit sha1_base64="ThBf6Fct1myVHSay2XFe4My8dic=">AAACunicdVFLbxMxEHaWVwmvFI5cLKJK5RLtFiQ4cKiokDgWqWkrZVcr25ndtWKvjT2LGpb9QfwarvTf4E2C1DQwkuXP33zz8Ay3SnqM4+tBdOfuvfsP9h4OHz1+8vTZaP/5uTeNEzAVRhl3yZkHJWuYokQFl9YB01zBBV+c9P6Lb+C8NPUZLi1kmpW1LKRgGKh8dJJqhhXn7acu/0pTBQXOwmVKavMUK0BGD9eSor3q6A/69/G9e01TJ8sKs3w0jifxyuguSDZgTDZ2mu8P8nRuRKOhRqGY97Mktpi1zKEUCrph2niwTCxYCbMAa6bBZ+3qtx09CMycFsaFUyNdsTcjWqa9X2oelH2v/ravJ//lmzVYvM9aWdsGoRbrQkWjKBraj47OpQOBahkAE06GXqmomGMCw4C3Mp0lWds316fZKs91NzygN5m+DYv6alvHVGlChUr/h5ZiN8B6aMJUzTwMMKwkub2AXXB+NEneTI6+vB0ff9wsZ4+8JK/IIUnIO3JMPpNTMiWC/CS/yG9yHX2IeCSjxVoaDTYxL8iWRfgHlrzdbQ==</latexit> <latexit sha1_base64="Min9X1kSrlMsgf0t/tlM/bkvIlw=">AAACo3icdVFNa9tAEF2rX4nTNk57zGWpMSRQjJQE2mNoL4Vc0mAnAUuI1WpkL9nVLrujYiP0J/prek3+Rf9NVo4hcdwOLDzevJl5O5MZKRyG4d9O8OLlq9dvtra7O2/fvd/t7X24dLqyHMZcS22vM+ZAihLGKFDCtbHAVCbhKrv53uavfoF1QpcjXBhIFJuWohCcoafS3udYMZxlRT1vaOyEoiaNEeZoVZ0zZA09eBQcpr1+OAyXQTdBtAJ9sorzdK+TxrnmlYISuWTOTaLQYFIzi4JLaLpx5cAwfsOmMPGwZApcUi+/1dCBZ3JaaOtfiXTJPq2omXJuoTKvbD2657mW/FduUmHxNalFaSqEkj8MKipJUdN2RzQXFjjKhQeMW+G9Uj5jlnH0m1zrNIqSujXXtlkbn6mmO6BPmdaGQTVf1zE51X7CTP2HFnyzwDio/FZ17hfoTxI9P8AmuDwaRsfDo58n/dNvq+NskX3yiRyQiHwhp+QHOSdjwslv8ofckrtgEJwFF8HoQRp0VjUfyVoEyT0JRtSg</latexit> Eq [log p✓ (x|z)] x ⇠ pdata (x) observed space (a) Computation Graph hierarchical hierarchical (b) Hierarchical dynamical dynamical (c) Sequential Figure 2.4: ELBO Computation Graphs. (a) Basic computation graph for varia7 tional inference. Outlined circles denote distributions. Smaller red circles denote 20 20 terms in the ELBO objective. Arrows, again, denote conditional dependencies. This notation can be used to express (b) hierarchical and (c) sequential models with various model dependencies. log-probability under the prior. The final term is the entropy of 𝑞(z|x), quantifying the spread or uncertainty of the distribution. Maximizing this term encourages 𝑞(z|x) to remain maximally uncertain, all else being equal (Jaynes, 1957). Often, the final two terms are combined, as in Eq. 2.22, with 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z)) interpreted as a form of “regularization,” penalizing deviations of 𝑞(z|x) from 𝑝 𝜃 (z). This KL divergence term has been the focus of recent investigations. In particular, one can interpret the ELBO as a Lagrangian, with Lagrange multiplier 𝛽 = 1: L (x; 𝑞, 𝜃) = Ez∼𝑞(z|x) [log 𝑝 𝜃 (x|z)] − 𝛽𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z)). (2.25) Thus, the ELBO can be seen as expressing a constrained optimization problem, with the regularizing constraint on 𝐷 KL (𝑞(z|x)|| 𝑝 𝜃 (z)) mediated by 𝛽. In models with flexible, e.g., autoregressive, conditional likelihoods, the model may not utilize z, in which case the KL term results in a local maximum at 𝑞(z|x) = 𝑝 𝜃 (z). This is the so-called “dying units” problem (Bowman et al., 2016), in which the latent variables are effectively unused. A common approach is to anneal 𝛽 from 0 → 1 during the course of training (Bowman et al., 2016; Sønderby et al., 2016), though alternative schemes exist (Kingma, Salimans, et al., 2016; X. Chen et al., 2017). We can also dynamically adjust 𝛽 to maintain a particular KL or conditional likelihood constraint (Rezende and Viola, 2018). And by setting 𝛽 > 1 we can over-regularize 𝑞(z|x), and with an independent prior, this can yield more disentangled latent variables (Higgins et al., 2017; Burgess et al., 2018). In these latter cases, where 𝛽 ≠ 1, L (x; 𝑞, 𝜃) is no longer a valid lower bound on log 𝑝 𝜃 (x). Alemi et al., 2018 provide 26 an information-theoretic framework to make sense of these ideas, identifying the connection between the two terms in the ELBO (Eq. 2.25) and the concepts of distortion and rate respectively. In short, adjusting 𝛽 allows one to target particular rate-distortion trade-offs, which, in turn, result in bounds on the mutual information between 𝑋 and 𝑍. This perspective allows one to consider latent variable models that achieve varying compression rates in complex, natural data domains, such as images (Ballé, Laparra, and Simoncelli, 2017) and video (Lombardo et al., 2019). As with autoregressive models (Figure 2.3), we can represent latent variable models and the ELBO objective as a computation graph, again breaking the dependency structure graphs from Figure 2.2 into separate distributions and terms in the objective. In Figure 2.4, we illustrate examples of these graphs. Each variable contains a red circle, denoting a term in the ELBO objective. In comparison with the fully-observed autoregressive model, we now have an additional objective term for the latent variable, corresponding to the KL divergence. This graphical representation also allows us to visualize the variational objective for more complex hierarchical and sequential models (Figures 2.4b & 2.4c). Inference Optimization In deriving variational inference and variational EM, we described inference optimization, the E-step, as the process of maximizing L (x; 𝑞, 𝜃) w.r.t. the distribution 𝑞(z|x). The fact that probability distributions are functions makes L (x; 𝑞, 𝜃) a functional and makes inference optimization a variational calculus problem. One could solve this optimization problem using a variety of techniques, ranging from gradient-free, e.g., evolution strategies, to first-order gradient-based, e.g., blackbox variational inference (Ranganath, Gerrish, and Blei, 2014), to second-order gradient-based, e.g., the Laplace approximation (Friston et al., 2007; Park, C. Kim, and G. Kim, 2019). Despite this range, first-order gradient-based techniques are most common in practice; when combined with amortization (see below), they can result in exceedingly efficient optimization. For this reason, we exclusively focus on first-order techniques in the following chapters, which we now describe in more detail. With a parametric 𝑞(z|x), first-order inference optimization entails calculating the gradients of L (x; 𝑞, 𝜃) w.r.t. the distribution parameters of 𝑞, which we refer to as   λ. For instance, if 𝑞(z|x) = N (z; µ𝑞 , diag(σ𝑞2 )), i.e., λ = µ𝑞 , σ𝑞 , then we must calculate ∇µ𝑞 L and ∇σ𝑞 L. We can consider the ELBO as an example of a stochastic 27 <latexit sha1_base64="HlSpur9H0Rx2BbfN6EKhfq3sCbs=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxWaEv6Awlk0nb0GQSkoxYhv6GW/0t/8ZMO4s+9ELgcO65uYdzQ8moNq77U3CKe/sHh6Wj8vHJ6dl5pXrR0yJRmHSxYEINQqQJozHpGmoYGUhFEA8Z6Yezl6zffydKUxF3zFySgKNJTMcUI2Mp3w956jMrj9BiVKm5DXdZcBd4OaiBvNqjamHkRwInnMQGM6T10HOlCVKkDMWMLMp+oolEeIYmZGhhjDjRQbo0vYB1y0RwLJR9sYFLdn0iRVzrOQ+tkiMz1du9jPyrN0zM+DFIaSwTQ2K8WjROGDQCZgnAiCqCDZtbgLCi1ivEU6QQNjanjZ86XpBm5rJvNtaHfFGuw3UmsyEN/9jUITYRdsOU/0NTvDsgNUlsqiKyAdqTeNsH2AW9ZsO7azTf7mut5/w4JXAFrsEN8MADaIFX0AZdgIEEn+ALfDtF59ZxneZK6hTymUuwUc7TL+fPxtw=</latexit> <latexit sha1_base64="HlSpur9H0Rx2BbfN6EKhfq3sCbs=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxWaEv6Awlk0nb0GQSkoxYhv6GW/0t/8ZMO4s+9ELgcO65uYdzQ8moNq77U3CKe/sHh6Wj8vHJ6dl5pXrR0yJRmHSxYEINQqQJozHpGmoYGUhFEA8Z6Yezl6zffydKUxF3zFySgKNJTMcUI2Mp3w956jMrj9BiVKm5DXdZcBd4OaiBvNqjamHkRwInnMQGM6T10HOlCVKkDMWMLMp+oolEeIYmZGhhjDjRQbo0vYB1y0RwLJR9sYFLdn0iRVzrOQ+tkiMz1du9jPyrN0zM+DFIaSwTQ2K8WjROGDQCZgnAiCqCDZtbgLCi1ivEU6QQNjanjZ86XpBm5rJvNtaHfFGuw3UmsyEN/9jUITYRdsOU/0NTvDsgNUlsqiKyAdqTeNsH2AW9ZsO7azTf7mut5/w4JXAFrsEN8MADaIFX0AZdgIEEn+ALfDtF59ZxneZK6hTymUuwUc7TL+fPxtw=</latexit> log q <latexit sha1_base64="Dq+cAEIkibq9J0Jy0Skhk1KMXas=">AAACjHicdVFdS8MwFM3q15w6p+KTL8Ex8Gm0U1AQYSiIjxP2BWspaZptYUlTk1QcpT/GV/1F/hvTbQ/70AuBw7n35B7ODWJGlbbtn4K1tb2zu1fcLx0cHpWPKyenXSUSiUkHCyZkP0CKMBqRjqaakX4sCeIBI71g8pT3e+9EKiqitp7GxONoFNEhxUgbyq+cu0yM4JufugFPXWaEIcoyv1K16/as4CZwFqAKFtXyTwq+GwqccBJpzJBSA8eOtZciqSlmJCu5iSIxwhM0IgMDI8SJ8tKZ/wzWDBPCoZDmRRrO2GVFirhSUx6YSY70WK33cvKv3iDRwzsvpVGcaBLh+aJhwqAWMA8DhlQSrNnUAIQlNV4hHiOJsDaRrfzUdrw0N5d/s7I+4FmpBpeZ3Eas+cfqHGIjYTaM+T80xZuCWJHEpCpCE6A5ibN+gE3QbdSd63rj9abafFwcpwguwCW4Ag64BU3wAlqgAzBIwSf4At9W2bqx7q2H+ahVWGjOwEpZz79GS8rt</latexit> z <latexit sha1_base64="JJBRHMAw1+FJL17Y9ikHFHSRDGA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuyIyaqDuiG5eY8IowIZ1OBxr6mLQdI074C7f6X/6NHWDBgN6kycm5j3N6bxAzqo3r/hScre2d3b3ifung8Oj4pFw57WiZKEzaWDKpegHShFFB2oYaRnqxIogHjHSDyVOW774RpakULTONic/RSNCIYmQs9TrgyIyDKP2YDctVt+7OA24CbwmqYBnNYaUwHIQSJ5wIgxnSuu+5sfFTpAzFjMxKg0STGOEJGpG+hQJxov10bnkGa5YJYSSVfcLAObvakSKu9ZQHtjKzqNdzGflXrp+Y6N5PqYgTQwReCEUJg0bC7P8wpIpgw6YWIKyo9QrxGCmEjd1SblLL89PMXDYmJx/wWakGV5nMRmz4e74OsZG0CmP+D03xZkOsSWK3KkO7QHsSb/0Am6BzXfdu6tcvt9XG4/I4RXAOLsAV8MAdaIBn0ARtgIEAn+ALfDsF59KpO+6i1Ckse85ALpyHX0lCxig=</latexit> ✏ <latexit sha1_base64="mlLz5PRn6k/2I/uXqfF1v+kMzi4=">AAACgnicdVFNS0JBFB1fVmZfWss2QyJEC3kvg1q0kNq0NMgK9CHzxqsOzVcz8yJ5+Dva1s/q3zRPXWTWhQuHc78O5yaaM+vC8KsQrBXXNzZLW+Xtnd29/Ur14MGq1FDoUMWVeUqIBc4kdBxzHJ60ASISDo/J801ef3wFY5mS926iIRZkJNmQUeI8FfcSkfVAW8aVnPYrtbARzgKvgmgBamgR7X610O8NFE0FSEc5sbYbhdrFGTGOUQ7Tci+1oAl9JiPoeiiJABtnM9VTXPfMAA+V8SkdnrE/JzIirJ2IxHcK4sb2dy0n/6p1Uze8jDMmdepA0vmhYcqxUzi3AA+YAer4xANCDfNaMR0TQ6jzRi1tuo/iLBeXr1k6n4hpuY5/MrkM7cTbch/hI+UvjMU/NKOrA9pC6l1VA2+gf0n0+wGr4OGsETUbZ3fntdb14jkldISO0QmK0AVqoVvURh1E0Qt6Rx/oMygGp0EUNOetQWExc4iWIrj6Bk+fx38=</latexit> z <latexit sha1_base64="JJBRHMAw1+FJL17Y9ikHFHSRDGA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuyIyaqDuiG5eY8IowIZ1OBxr6mLQdI074C7f6X/6NHWDBgN6kycm5j3N6bxAzqo3r/hScre2d3b3ifung8Oj4pFw57WiZKEzaWDKpegHShFFB2oYaRnqxIogHjHSDyVOW774RpakULTONic/RSNCIYmQs9TrgyIyDKP2YDctVt+7OA24CbwmqYBnNYaUwHIQSJ5wIgxnSuu+5sfFTpAzFjMxKg0STGOEJGpG+hQJxov10bnkGa5YJYSSVfcLAObvakSKu9ZQHtjKzqNdzGflXrp+Y6N5PqYgTQwReCEUJg0bC7P8wpIpgw6YWIKyo9QrxGCmEjd1SblLL89PMXDYmJx/wWakGV5nMRmz4e74OsZG0CmP+D03xZkOsSWK3KkO7QHsSb/0Am6BzXfdu6tcvt9XG4/I4RXAOLsAV8MAdaIBn0ARtgIEAn+ALfDsF59KpO+6i1Ckse85ALpyHX0lCxig=</latexit> ⇤ <latexit sha1_base64="Nkk3T+niU2cCEWunvav3KuI5f5w=">AAACdnicdVHLSgMxFE3HV62vVpeCBEtRXJSZKuiy6MZlC31BO5RMetuGJpMhyYhl6Be41Y/zT1yaabvoQy8EDuc+zsm9QcSZNq77nXF2dvf2D7KHuaPjk9OzfOG8pWWsKDSp5FJ1AqKBsxCahhkOnUgBEQGHdjB5SfPtN1CaybBhphH4goxCNmSUGEvV7/r5olt254G3gbcERbSMWr+Q6fcGksYCQkM50brruZHxE6IMoxxmuV6sISJ0QkbQtTAkArSfzJ3OcMkyAzyUyr7Q4Dm72pEQofVUBLZSEDPWm7mU/CvXjc3wyU9YGMUGQroQGsYcG4nTb+MBU0ANn1pAqGLWK6Zjogg1djlrkxqen6Tm0jFr8oGY5Up4lUltREa8r9cRPpJWYSz+oRndbog0xHarcmAXaE/ibR5gG7QqZe++XKk/FKvPy+Nk0SW6RrfIQ4+oil5RDTURRYA+0Cf6yvw4V07JuVmUOpllzwVaC8f9BZZ5wrc=</latexit> (a) Inference (b) Score Function (c) Pathwise Derivative Figure 2.5: Stochastic Gradient Estimation. (a) Variational inference with parametric approximate posteriors requires optimizing L w.r.t. the distribution parameters, λ. This often requires estimating stochastic gradients (red dotted lines). (b) The score function estimator (Eq. 2.26) is applicable to any distribution, but suffers from high variance. 8 Note: the solid red arrow denotes the per-sample objective, 𝑙 (x, z; 𝜃). 9 (c) The pathwise derivative estimator 10 (Eq. 2.27), in contrast, has lower variance, but is less widely applicable. computation graph (Schulman et al., 2015), containing the stochastic sampling operation z ∼ 𝑞(z|x). Unfortunately, stochastic sampling is non-differentiable, preventing us from simply differentiating through the chain λ → z → L. Schulman et al., 2015 highlight two stochastic gradient estimators to tackle this issue: the score function estimator (Glynn, 1990; Williams, 1992; Fu, 2006) and the pathwise derivative estimator (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014; Titsias and Lázaro-Gredilla, 2014), each of which allows us to calculate stochastic estimates of ∇λ L. The score function estimator, sometimes referred to as the REINFORCE estimator (Williams, 1992), is generally applicable to any parametric distribution. With 𝑞 λ (z|x) denoting the dependence of 𝑞 on λ, as well as L (x; 𝑞, 𝜃) = Ez∼𝑞λ (z|x) [𝑙 (x, z; 𝜃)], the score function estimator allows us to express ∇λ L as ∇λ Ez∼𝑞λ (z|x) [𝑙 (x, z; 𝜃)] = Ez∼𝑞λ (z|x) [∇λ log 𝑞 λ (z|x)𝑙 (x, z; 𝜃)] . (2.26) While the score function estimator is unbiased and can be successfully utilized for variational inference (Gregor et al., 2014; Ranganath, Gerrish, and Blei, 2014; Mnih and Gregor, 2014; Mnih and Rezende, 2016), in practice, it requires considerable tuning and computation, owing to its empirically high variance. Alternatively, the pathwise derivative estimator, sometimes referred to as the reparameterization estimator (Kingma and Welling, 2014), when applicable, allows us to obtain unbiased 28 gradient estimates with considerably lower variance. This is accomplished by reparameterizing z in terms of an auxiliary random variable, enabling differentiation through stochastic sampling. The most common example is reparameterizing z ∼ N (z; µ𝑞 , diag(σ𝑞2 )) as z = µ𝑞 +  σ𝑞 , where  ∼ N (; 0, I) and denotes element-wise multiplication. More generally, for some deterministic function 𝑔, if we can express z = 𝑔(λ, ) with  ∼ 𝑝(), then the pathwise derivative estimator allows us to express ∇λ L as ∇λ Ez∼𝑞λ (z|x) [𝑙 (x, z; 𝜃)] = E∼𝑝() [∇λ 𝑙 (x, 𝑔(λ, ); 𝜃)] . (2.27) With these stochastic gradient estimators in hand, we can perform stochastic gradientbased optimization of L (x; 𝑞, 𝜃) w.r.t. λ (Ranganath, Gerrish, and Blei, 2014): estimating ∇λ L by sampling z (or ) and updating λ using stochastic gradient ascent. Unfortunately, a naïve implementation of this inference optimization (Estep) procedure scales poorly to large models and data sets. This is because many gradient steps must be performed per example, just to provide one estimate of L (x; 𝑞, 𝜃) for M-step optimization. More sophisticated approaches cache previous estimates of λ for each example to initialize future inference optimization. However, beyond the additional memory overhead, this approach still requires tuning the inference optimizer’s learning rate and is not applicable to online learning settings. In order to apply variational inference more broadly, we require a simple technique for dramatically improving the efficiency of inference optimization. Amortization (Gershman and Goodman, 2014), a form of meta-optimization, offers a viable solution. Amortized Variational Inference Amortization, in a generic sense, refers to spreading out costs. In amortized inference, these “costs” are the computational costs of performing inference optimization. Thus, rather than separately optimizing λ for each data example, we amortize this optimization cost using a learned optimizer, i.e., an inference model. By using this meta-optimization procedure, we can perform inference optimization far more efficiently for each example, with a negligible cost for learning the inference model. The concept of inference models is deeply embedded with deep latent variable models, popularized by the Helmholtz Machine (Dayan et al., 1995), which was formulated as an autoencoder (Ballard, 1987). Formally, in such setups, the inference model is a direct mapping from x to λ: λ ← 𝑓𝜙 (x), (2.28) 29 (x)⇤ <latexit sha1_base64="Nkk3T+niU2cCEWunvav3KuI5f5w=">AAACdnicdVHLSgMxFE3HV62vVpeCBEtRXJSZKuiy6MZlC31BO5RMetuGJpMhyYhl6Be41Y/zT1yaabvoQy8EDuc+zsm9QcSZNq77nXF2dvf2D7KHuaPjk9OzfOG8pWWsKDSp5FJ1AqKBsxCahhkOnUgBEQGHdjB5SfPtN1CaybBhphH4goxCNmSUGEvV7/r5olt254G3gbcERbSMWr+Q6fcGksYCQkM50brruZHxE6IMoxxmuV6sISJ0QkbQtTAkArSfzJ3OcMkyAzyUyr7Q4Dm72pEQofVUBLZSEDPWm7mU/CvXjc3wyU9YGMUGQroQGsYcG4nTb+MBU0ANn1pAqGLWK6Zjogg1djlrkxqen6Tm0jFr8oGY5Up4lUltREa8r9cRPpJWYSz+oRndbog0xHarcmAXaE/ibR5gG7QqZe++XKk/FKvPy+Nk0SW6RrfIQ4+oil5RDTURRYA+0Cf6yvw4V07JuVmUOpllzwVaC8f9BZZ5wrc=</latexit> <latexit sha1_base64="byTPipI6nSdWcCa+1Ot0FEheE08=">AAAClHicdVHLSgMxFE3HV62vquBGhGARdFNmVNCFi2IRXEmFVoXOMGTSTBtMJiHJiGWYlV/jVr/GvzHTdtGHXggczn2dnBtJRrVx3Z+Ss7S8srpWXq9sbG5t71R39560SBUmHSyYUC8R0oTRhHQMNYy8SEUQjxh5jl6bRf75jShNRdI2Q0kCjvoJjSlGxlJh9ciPeOZr2ucoD305oPDU58gMojh7z8/Cas2tu6OAi8CbgBqYRCvcLYV+T+CUk8RghrTueq40QYaUoZiRvOKnmkiEX1GfdC1MECc6yEb/yOGJZXowFsq+xMARO92RIa71kEe2stCo53MF+Veum5r4OshoIlNDEjxeFKcMGgELU2CPKoING1qAsKJWK8QDpBA21rqZSW0vyApxxZiZ9RHPKydwmilkSMPfZ+sQ6wu7YcD/oSlebJCapNZV0bMG2pN48wdYBE/nde+ifv54WWvcTo5TBofgGJwCD1yBBrgHLdABGHyAT/AFvp0D58ZpOnfjUqc06dkHM+E8/AKgM84S</latexit> ✏ <latexit sha1_base64="mlLz5PRn6k/2I/uXqfF1v+kMzi4=">AAACgnicdVFNS0JBFB1fVmZfWss2QyJEC3kvg1q0kNq0NMgK9CHzxqsOzVcz8yJ5+Dva1s/q3zRPXWTWhQuHc78O5yaaM+vC8KsQrBXXNzZLW+Xtnd29/Ur14MGq1FDoUMWVeUqIBc4kdBxzHJ60ASISDo/J801ef3wFY5mS926iIRZkJNmQUeI8FfcSkfVAW8aVnPYrtbARzgKvgmgBamgR7X610O8NFE0FSEc5sbYbhdrFGTGOUQ7Tci+1oAl9JiPoeiiJABtnM9VTXPfMAA+V8SkdnrE/JzIirJ2IxHcK4sb2dy0n/6p1Uze8jDMmdepA0vmhYcqxUzi3AA+YAer4xANCDfNaMR0TQ6jzRi1tuo/iLBeXr1k6n4hpuY5/MrkM7cTbch/hI+UvjMU/NKOrA9pC6l1VA2+gf0n0+wGr4OGsETUbZ3fntdb14jkldISO0QmK0AVqoVvURh1E0Qt6Rx/oMygGp0EUNOetQWExc4iWIrj6Bk+fx38=</latexit> µ (x)+ <latexit sha1_base64="Z+9aU0F2yHtHC/FjEFUaW/dEtUg=">AAACdnicdVHLSgMxFE3HV62vVpeCBEtREMpMFXRZdOOyhb6gHUomvW1Dk8mQZMQy9Avc6sf5Jy7NtF30oRcCh3Mf5+TeIOJMG9f9zjg7u3v7B9nD3NHxyelZvnDe0jJWFJpUcqk6AdHAWQhNwwyHTqSAiIBDO5i8pPn2GyjNZNgw0wh8QUYhGzJKjKXqd/180S2788DbwFuCIlpGrV/I9HsDSWMBoaGcaN313Mj4CVGGUQ6zXC/WEBE6ISPoWhgSAdpP5k5nuGSZAR5KZV9o8Jxd7UiI0HoqAlspiBnrzVxK/pXrxmb45CcsjGIDIV0IDWOOjcTpt/GAKaCGTy0gVDHrFdMxUYQau5y1SQ3PT1Jz6Zg1+UDMciW8yqQ2IiPe1+sIH0mrMBb/0IxuN0QaYrtVObALtCfxNg+wDVqVsndfrtQfitXn5XGy6BJdo1vkoUdURa+ohpqIIkAf6BN9ZX6cK6fk3CxKncyy5wKtheP+ApiLwrg=</latexit> <latexit sha1_base64="f4POFtkrBQtlxidt2haan9e3VOs=">AAACkXicdVHLSgMxFE3HV62vane6CRZBN2WmCuqu6EZwo9AXdIYhk2baYDIJSUYsw4Bf41Z/x78xU2dhrV4IHM59nZwbSUa1cd3PirOyura+Ud2sbW3v7O7V9w/6WqQKkx4WTKhhhDRhNCE9Qw0jQ6kI4hEjg+jptsgPnonSVCRdM5Mk4GiS0JhiZCwV1g/9iGc+T/PQl1MKT32OzDSKs5f8LKw33ZY7D7gMvBI0QRkP4X4l9McCp5wkBjOk9chzpQkypAzFjOQ1P9VEIvyEJmRkYYI40UE2/0QOTywzhrFQ9iUGztmfHRniWs94ZCsLjfp3riD/yo1SE18FGU1kakiCvxfFKYNGwMIROKaKYMNmFiCsqNUK8RQphI31bWFS1wuyQlwxZmF9xPPaCfzJFDKk4S+LdYhNhN0w5f/QFC83SE1S66oYWwPtSbzfB1gG/XbLO2+1Hy+anZvyOFVwBI7BKfDAJeiAO/AAegCDV/AG3sGH03CunY5T1jqVsqcBFsK5/wKbEczF</latexit> z <latexit sha1_base64="JJBRHMAw1+FJL17Y9ikHFHSRDGA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuyIyaqDuiG5eY8IowIZ1OBxr6mLQdI074C7f6X/6NHWDBgN6kycm5j3N6bxAzqo3r/hScre2d3b3ifung8Oj4pFw57WiZKEzaWDKpegHShFFB2oYaRnqxIogHjHSDyVOW774RpakULTONic/RSNCIYmQs9TrgyIyDKP2YDctVt+7OA24CbwmqYBnNYaUwHIQSJ5wIgxnSuu+5sfFTpAzFjMxKg0STGOEJGpG+hQJxov10bnkGa5YJYSSVfcLAObvakSKu9ZQHtjKzqNdzGflXrp+Y6N5PqYgTQwReCEUJg0bC7P8wpIpgw6YWIKyo9QrxGCmEjd1SblLL89PMXDYmJx/wWakGV5nMRmz4e74OsZG0CmP+D03xZkOsSWK3KkO7QHsSb/0Am6BzXfdu6tcvt9XG4/I4RXAOLsAV8MAdaIBn0ARtgIEAn+ALfDsF59KpO+6i1Ckse85ALpyHX0lCxig=</latexit> Figure 2.6: Variational Autoencoder (VAE). VAEs combine direct amortization (Eq. 2.28) and the pathwise derivative estimator (Eq. 2.27, top) with Gaussian approximate posteriors to train deep latent variable models. In the model diagram (center), the amortized inference model (left) acts as an encoder, with the conditional likelihood (right) acting as a decoder. Each are parameterized by deep networks. 35 where 𝑓𝜙 is a model (deep network) with parameters 𝜙. Conventionally, we denote the approximate posterior as 𝑞 𝜙 (z|x) to denote the parameterization by 𝜙. Now, rather than optimizing λ using gradient-based techniques, we periodically update 𝜕λ 𝜙 using ∇𝜙 L = 𝜕L 𝜕 λ 𝜕𝜙 , thereby letting 𝑓 𝜙 learn to optimize λ. This procedure is incredibly simple, as we only need to tune the learning rate for 𝜙, and efficient, as we have an estimate of λ after only one forward pass through 𝑓𝜙 . Amortization is also widely applicable: if we can estimate ∇λ L using stochastic gradient estimation (see above), we can continue differentiating through the chain 𝜙 → λ → z → L. When direct amortization is combined with the pathwise derivative estimator in deep latent variable models, the resulting setup is referred to as a variational autoencoder (VAE) (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). In this autoencoder interpretation, 𝑞 𝜙 (z|x) is an encoder, z is the latent code, and 𝑝 𝜃 (x|z) is a decoder. A computation graph is shown in Figure 2.6. This direct encoding scheme seems intuitively obvious: in the same way that 𝑝 𝜃 (x|z) directly maps z to a distribution over x, 𝑞 𝜙 (z|x) directly maps x to a distribution over z. Indeed, with perfect knowledge of 𝑝 𝜃 (x, z), 𝑓𝜙 could act as a lookup table, precisely mapping each x to the corresponding optimal λ. However, as we discuss in Chapter 3, there are a number of subtle issues that can arise with direct amortization, forming one of 30 the primary motivations for the following chapters. 2.4 Discussion In this chapter, we have reviewed the basic considerations in formulating and training probabilistic models. We started by introducing the maximum log-likelihood (or minimum KL divergence) framework for model training, which involves maximizing Ex∼𝑝data (x) [log 𝑝 𝜃 (x)] w.r.t. the model parameters, 𝜃. The model is formulated in terms of the dependency structure between variables, the distribution families, and the functional form of the distribution parameters. Each of these constitute design choices, which are often selected in light of known structure in the data. In improving the expressive capacity of the model, we discussed two primary methods for adding dependency structure to the model, autoregression and latent variables, as well as combinations of the two. While we presented these ideas separately, there is, in fact, a link between these two forms of dependency structure. In particular, Kingma, Salimans, et al., 2016 noted that we can express sampling from a Gaussian autoregressive model in both an autoregressive form: 𝑥 𝑗 ∼ N (𝑥 𝑗 ; 𝜇𝜃 (𝑥 < 𝑗 ), 𝜎𝜃2 (𝑥 < 𝑗 )), (2.29) as well as in the form of an affine flow-based latent variable model: 𝑥 𝑗 = 𝜇𝜃 (𝑥 < 𝑗 ) + 𝜎𝜃 (𝑥 < 𝑗 ) · 𝑧 𝑗 , (2.30) where 𝑧 𝑗 ∼ N (𝑧 𝑗 ; 0, 1) is an auxiliary latent variable. Note that Eq. 2.30 is the reparameterization trick (from the pathwise derivative estimator) applied to the Gaussian autoregressive sampling in Eq. 2.29. To train an autoregressive model, we can analytically evaluate the log-density directly via Eq. 2.29. Alternatively, we can evaluate the log-density by performing “inference,” inverting the affine transform: 𝑧𝑗 = 𝑥 𝑗 − 𝜇𝜃 (𝑥 < 𝑗 ) , 𝜎𝜃 (𝑥 < 𝑗 ) (2.31) and then evaluating the log-density of 𝑧 𝑗 under 𝑝 𝜃 (𝑧) = N (0, 1) and the log-scaling factor of the transform, log 𝜎𝜃 (𝑥 < 𝑗 ). With this scheme, we exactly recover the Gaussian density definition in Eq. 2.12. This case of equivalence between particular classes of autoregressive models and (flow-based) latent variable models suggests that, rather than considering various model classes as entirely distinct, we can consider the general computational principles of probability, with various models making trade-offs in terms of tractability, expressive capacity, computational efficiency, etc. 31 This chapter has not exhaustively covered the breadth of probabilistic modeling techniques that exist in the literature, instead focusing on the subset of ideas relevant to this thesis. It is worth mentioning that a variety of techniques have been recently developed or have gained renewed interest. For instance, implicit latent variable models (Mohamed and Lakshminarayanan, 2016), such as generative adversarial networks (Goodfellow, Pouget-Abadie, et al., 2014) and generative stochastic networks (Y. Bengio, Laufer, et al., 2014), are formulated in terms of a sampling procedure, avoiding the need to explicitly define 𝑝 𝜃 (x). Energy-based models (LeCun et al., 2006; Salakhutdinov and G. Hinton, 2009; Du and Mordatch, 2019) similarly offer a more flexible form, parameterizing an energy function over the space of x, which is then converted into a Boltzmann distribution. Score function models (Hyvärinen and Dayan, 2005; Vincent, 2011; Song and Ermon, 2019) avoid directly parameterizing the energy function, instead estimating a gradient field to sample from the data distribution. Finally, a variety of contrastive learning procedures have been recently developed (Gutmann and Hyvärinen, 2010; Mnih and Kavukcuoglu, 2013; A. v. d. Oord, Li, and Vinyals, 2018), providing an alternative, “self-supervised” technique for modeling data while avoiding explicit data density estimation. These ideas may seem hopelessly diverse, however, they are all tied to concepts in probabilistic modeling (and by extension, information theory), as well as learning, inference, and optimization more generally. Indeed, as with the example of flow-based latent variable models and autoregressive models discussed above, many modeling approaches can be interpreted as related or special cases of each other, or even combined in new and interesting ways (Rezende and Mohamed, 2015; Makhzani et al., 2015; Agrawal and Dukkipati, 2016). Thus, the advances in probabilistic modeling and inference presented in this thesis should not be viewed in isolation, as their applicability may cross the blurred boundaries between different approaches. References Agrawal, Siddharth and Ambedkar Dukkipati (2016). “Deep Variational Inference Without Pixel-Wise Reconstruction”. In: arXiv preprint arXiv:1611.05209. Alemi, Alexander et al. (2018). “Fixing a Broken ELBO”. In: International Conference on Machine Learning, pp. 159–168. Ballard, Dana H (1987). “Modular Learning in Neural Networks.” In: AAAI, pp. 279– 284. 32 Ballé, Johannes, Valero Laparra, and Eero P Simoncelli (2017). “End-to-end optimized image compression”. In: International Conference on Learning Representations. Bell, Anthony J and Terrence J Sejnowski (1997). “The “independent components” of natural scenes are edge filters”. In: Vision research 37.23, pp. 3327–3338. Bengio, Yoshua and Samy Bengio (2000). “Modeling high-dimensional discrete data with multi-layer neural networks”. In: Advances in Neural Information Processing Systems, pp. 400–406. Bengio, Yoshua, Eric Laufer, et al. (2014). “Deep generative stochastic networks trainable by backprop”. In: International Conference on Machine Learning. PMLR, pp. 226–234. Bowman, Samuel R et al. (2016). “Generating Sentences from a Continuous Space”. In: CoNLL 2016, p. 10. Burgess, Christopher P et al. (2018). “Understanding disentangling in 𝛽-VAE”. In: arXiv preprint arXiv:1804.03599. Chen, Scott Saobing and Ramesh A Gopinath (2001). “Gaussianization”. In: Advances in neural information processing systems, pp. 423–429. Chen, Xi et al. (2017). “Variational Lossy Autoencoder”. In: International Conference on Learning Representations. Chua, Kurtland et al. (2018). “Deep reinforcement learning in a handful of trials using probabilistic dynamics models”. In: Advances in Neural Information Processing Systems, pp. 4754–4765. Cornish, Rob et al. (2020). “Relaxing bijectivity constraints with continuously indexed normalising flows”. In: International Conference on Machine Learning. Cvitkovic, Milan and Günther Koliander (2019). “Minimal Achievable Sufficient Statistic Learning”. In: International Conference on Machine Learning, pp. 1465– 1474. Dayan, Peter et al. (1995). “The helmholtz machine”. In: Neural computation 7.5, pp. 889–904. Dempster, Arthur P, Nan M Laird, and Donald B Rubin (1977). “Maximum likelihood from incomplete data via the EM algorithm”. In: Journal of the royal statistical society. Series B (methodological), pp. 1–38. Dinh, Laurent, David Krueger, and Yoshua Bengio (2015). “Nice: Non-linear independent components estimation”. In: International Conference on Learning Representations. Du, Yilun and Igor Mordatch (2019). “Implicit Generation and Modeling with Energy Based Models”. In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc. 33 Efron, Bradley and Carl Morris (1973). “Stein’s estimation rule and its competitors—an empirical Bayes approach”. In: Journal of the American Statistical Association 68.341, pp. 117–130. Frey, Brendan J, Geoffrey E Hinton, and Peter Dayan (1996). “Does the wake-sleep algorithm produce good density estimators?” In: Advances in neural information processing systems, pp. 661–667. Friston, Karl et al. (2007). “Variational free energy and the Laplace approximation”. In: Neuroimage 34.1, pp. 220–234. Fu, Michael C (2006). “Gradient estimation”. In: Handbooks in operations research and management science 13, pp. 575–616. Gershman, Samuel and Noah Goodman (2014). “Amortized inference in probabilistic reasoning”. In: Proceedings of the Cognitive Science Society. Vol. 36. 36. Glynn, Peter W (1990). “Likelihood ratio gradient estimation for stochastic systems”. In: Communications of the ACM 33.10, pp. 75–84. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville (2016). Deep learning. MIT press. Goodfellow, Ian, Jean Pouget-Abadie, et al. (2014). “Generative Adversarial Nets”. In: Advances in Neural Information Processing Systems 27, pp. 2672–2680. Graves, Alex (2013). “Generating sequences with recurrent neural networks”. In: arXiv preprint arXiv:1308.0850. Gregor, Karol et al. (2014). “Deep autoregressive networks”. In: Proceedings of the International Conference on Machine Learning (ICML), pp. 1242–1250. Gulrajani, Ishaan et al. (2017). “Pixelvae: A latent variable model for natural images”. In: International Conference on Learning Representations. Gutmann, Michael and Aapo Hyvärinen (2010). “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models”. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, pp. 297–304. Ha, David and Jürgen Schmidhuber (2018). “Recurrent world models facilitate policy evolution”. In: Advances in Neural Information Processing Systems, pp. 2450– 2462. Higgins, Irina et al. (2017). “beta-vae: Learning basic visual concepts with a constrained variational framework”. In: Proceedings of the International Conference on Learning Representations (ICLR). Hinton, Geoffrey E and Drew Van Camp (1993). “Keeping the neural networks simple by minimizing the description length of the weights”. In: Proceedings of the sixth annual conference on Computational learning theory. ACM, pp. 5–13. 34 Hoffman, Matthew D et al. (2013). “Stochastic variational inference”. In: The Journal of Machine Learning Research 14.1, pp. 1303–1347. Hyvärinen, Aapo and Peter Dayan (2005). “Estimation of non-normalized statistical models by score matching.” In: Journal of Machine Learning Research 6.4. Hyvärinen, Aapo and Erkki Oja (2000). “Independent component analysis: algorithms and applications”. In: Neural networks 13.4-5, pp. 411–430. Jaynes, Edwin T (1957). “Information theory and statistical mechanics”. In: Physical review 106.4, p. 620. Jordan, Michael I et al. (1998). “An introduction to variational methods for graphical models”. In: NATO ASI SERIES D BEHAVIOURAL AND SOCIAL SCIENCES 89, pp. 105–162. Kingma, Durk P, Tim Salimans, et al. (2016). “Improved variational inference with inverse autoregressive flow”. In: Advances in neural information processing systems, pp. 4743–4751. Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Kumar, Manoj et al. (2020). “VideoFlow: A Flow-Based Generative Model for Video”. In: International Conference on Learning Representations. Laparra, Valero, Gustavo Camps-Valls, and Jesús Malo (2011). “Iterative gaussianization: from ICA to random rotations”. In: IEEE transactions on neural networks 22.4, pp. 537–549. LeCun, Yann et al. (2006). “A tutorial on energy-based learning”. In: Predicting structured data 1.0. Lombardo, Salvator et al. (2019). “Deep Generative Video Compression”. In: Advances in Neural Information Processing Systems, pp. 9283–9294. Makhzani, Alireza et al. (2015). “Adversarial autoencoders”. In: arXiv preprint arXiv:1511.05644. Mnih, Andriy and Karol Gregor (2014). “Neural Variational Inference and Learning in Belief Networks”. In: International Conference on Machine Learning, pp. 1791– 1799. Mnih, Andriy and Koray Kavukcuoglu (2013). “Learning word embeddings efficiently with noise-contrastive estimation”. In: Advances in neural information processing systems 26, pp. 2265–2273. Mnih, Andriy and Danilo Jimenez Rezende (2016). “Variational Inference for Monte Carlo Objectives”. In: International Conference on Machine Learning, pp. 2188– 2196. 35 Mohamed, Shakir and Balaji Lakshminarayanan (2016). “Learning in implicit generative models”. In: arXiv preprint arXiv:1610.03483. Murphy, Kevin P (2012). Machine learning: a probabilistic perspective. MIT press. Neal, Radford M and Geoffrey E Hinton (1998). “A view of the EM algorithm that justifies incremental, sparse, and other variants”. In: Learning in graphical models. Springer, pp. 355–368. Olshausen, Bruno A and David J Field (1996). “Emergence of simple-cell receptive field properties by learning a sparse code for natural images”. In: Nature 381.6583, p. 607. Oord, Aaron van den, Yazhe Li, and Oriol Vinyals (2018). “Representation learning with contrastive predictive coding”. In: arXiv preprint arXiv:1807.03748. Oord, Aäron van den et al. (2016). “WaveNet: A Generative Model for Raw Audio”. In: 9th ISCA Speech Synthesis Workshop, pp. 125–125. Park, Yookoon, Chris Kim, and Gunhee Kim (2019). “Variational Laplace Autoencoders”. In: International Conference on Machine Learning, pp. 5032–5041. Radford, Alec et al. (2019). “Language Models are Unsupervised Multitask Learners”. In: Ranganath, Rajesh, Sean Gerrish, and David Blei (2014). “Black box variational inference”. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 814–822. Razavi, Ali, Aaron van den Oord, and Oriol Vinyals (2019). “Generating diverse highfidelity images with vq-vae-2”. In: Advances in Neural Information Processing Systems, pp. 14866–14876. Rezende, Danilo Jimenez and Shakir Mohamed (2015). “Variational Inference with Normalizing Flows”. In: International Conference on Machine Learning, pp. 1530–1538. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Rezende, Danilo Jimenez and Fabio Viola (2018). “Taming VAEs”. In: arXiv preprint arXiv:1810.00597. Rippel, Oren and Ryan Prescott Adams (2013). “High-dimensional probability estimation with deep density models”. In: arXiv preprint arXiv:1302.5125. Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). “Learning representations by back-propagating errors.” In: Nature. Salakhutdinov, Ruslan and Geoffrey Hinton (2009). “Deep boltzmann machines”. In: Artificial intelligence and statistics. PMLR, pp. 448–455. 36 Schulman, John et al. (2015). “Gradient estimation using stochastic computation graphs”. In: Advances in Neural Information Processing Systems, pp. 3528–3536. Sønderby, Casper Kaae et al. (2016). “Ladder variational autoencoders”. In: Advances in Neural Information Processing Systems (NIPS), pp. 3738–3746. Song, Yang and Stefano Ermon (2019). “Generative modeling by estimating gradients of the data distribution”. In: arXiv preprint arXiv:1907.05600. Sutskever, Ilya, Oriol Vinyals, and Quoc V Le (2014). “Sequence to sequence learning with neural networks”. In: Advances in neural information processing systems, pp. 3104–3112. Tabak, Esteban G and Cristina V Turner (2013). “A family of nonparametric density estimation algorithms”. In: Communications on Pure and Applied Mathematics 66.2, pp. 145–164. Tipping, Michael E and Christopher M Bishop (1999). “Probabilistic principal component analysis”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61.3, pp. 611–622. Titsias, Michalis and Miguel Lázaro-Gredilla (2014). “Doubly stochastic variational Bayes for non-conjugate inference”. In: International conference on machine learning, pp. 1971–1979. Vincent, Pascal (2011). “A connection between score matching and denoising autoencoders”. In: Neural computation 23.7, pp. 1661–1674. Williams, Ronald J (1992). “Simple statistical gradient-following algorithms for connectionist reinforcement learning”. In: Reinforcement Learning. Springer, pp. 5–32. Part II Perception 37 38 Chapter 3 ITERATIVE AMORTIZED INFERENCE learned negative feedback (static) perception Marino, Joseph, Yisong Yue, and Stephan Mandt (2018). “Iterative Amortized 9 Inference”. In: International Conference on Machine Learning, pp. 3403–3412. url: http://proceedings.mlr.press/v80/marino18a.html. 3.1 Introduction In Chapter 2, we described latent variable models, 𝑝 𝜃 (x, z) = 𝑝 𝜃 (x|z) 𝑝 𝜃 (z), defined over observed variables, x, and latent variables, z. If we take the observed variables to be some form of sensory observation, e.g., visual, audio, or tactile input, then we can reasonably describe the compressed representation in z as a state estimate. That is, if we treat 𝑝 𝜃 (x, z) as an internal “forward model” of the environment generating the observations, i.e., the generative process, then z corresponds to the underlying state of the environment. Note that this does not necessarily correspond to the actual state of the environment; indeed, z may not be interpretable at all. Given an observation, we will refer to the process of state estimation, i.e., inferring 𝑝 𝜃 (z|x) or some approximation 𝑞(z|x), as perception. As noted in Chapter 2, this is generally computationally intractable, as it requires marginalizing over z. For this reason, we described the technique of variational inference (Section 2.3), formulating inference as optimization. Finally, we noted that the concept of amortization has proven essential in efficiently performing variational inference in deep generative models, most notably in variational autoencoders (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). 39 However, while amortization has dramatically improved the efficiency of variational inference, the direct inference models (or “encoders”) typically employed raise their own issues (see Section 3.2). This chapter attempts to overcome these issues by introducing a more flexible and general amortization technique referred to as iterative amortized inference. Broadly, rather than framing variational inference as a direct mapping from observations to approximate posterior estimates, we use gradients or errors to iteratively update these estimates. In this way, inference once again becomes an iterative optimization algorithm, using negative feedback to minimize errors or gradient magnitudes, but now with the efficiency of amortization. This idea was inspired by (hierarchical) predictive coding (Mumford, 1992; Rao and Ballard, 1999; Friston, 2005), which uses weighted errors to perform inference. In turn, these works borrowed heavily from classical Bayesian inference techniques, such as Kalman filtering (Kalman, 1960), which performs exact probabilistic inference using prediction errors. More recent works, discussed below, have pursued similar ideas. 3.2 Issues with Direct Inference Models The Amortization Gap Variational inference reformulates inference as the maximization of L w.r.t. 𝑞(z|x), constituting the expectation step of the variational EM algorithm (Algorithm 1). In general, this is a difficult non-convex optimization problem, typically requiring a lengthy iterative estimation procedure (Ranganath, Gerrish, and Blei, 2014). Yet, direct inference models attempt to perform this optimization through a direct, discriminative mapping from data observations to approximate posterior parameters. Of course, generative models can adapt to accommodate sub-optimal approximate posteriors. Nevertheless, the possible limitations of a direct inference mapping applied to this difficult optimization procedure may result in sub-optimal estimates, limiting model performance. We demonstrate this concept in Figure 3.1 by visualizing the optimization surface of L defined by a 2D latent Gaussian model and a particular binarized MNIST (LeCun et al., 1998) data example. To visualize the approximate posterior, we use a point estimate, 𝑞(z|x) = 𝛿(µ𝑞 ), where µ𝑞 = (𝜇1 , 𝜇2 ) is the estimate and 𝛿 is the Dirac delta function. See Appendix C.1 for details. Shown on the plot are the optimal (maximum a posteriori or MAP) estimate, the estimate from a direct inference model, and an optimization trajectory of gradient ascent. The inference model is unable to achieve the optimum, but manages to output a reasonable estimate in one forward pass. Gradient ascent requires many iterations and is sensitive to step-size, but FEEDBACK PERCEPTION direct inference models provide suboptimal estimates 40 “amortization gap” Direct Inference Model <latexit sha1_base64="LhH3Wz9gi4TKq8xznA9WRXhble8=">AAAClnicdVFbSxtBFJ6srZe02mhfCj44GIQ+hd1YsE9FqlJ9EFIwKiRLmJ2cTQbnssycLYZlH/01vtYf479xNubBGD1w4OM79/MlmRQOw/CxFix9+Li8srpW//R5feNLY3Pr0pnccuhyI429TpgDKTR0UaCE68wCU4mEq+TmqIpf/QPrhNEXOMkgVmykRSo4Q08NGjt9hFu0qjgWFjjSM52CBc2BnpshyHLQaIatcGp0EUQz0CQz6ww2a4P+0PBcgUYumXO9KMwwLphFwSWU9X7uIGP8ho2g56FmClxcTC8p6Z5nhjQ11rtGOmVfVhRMOTdRic9UDMfudawi34r1ckx/xoXQWY7+uudBaS4pGlq9hQ6n58uJB4xb4XelfMws4+ifN9fpIoqLarmqzdz4RJX1PfqSqdbIUN3O5zE5Mn7CWL1DC75YkDnI/Ve9JGXdSxK9FmARXLZb0X6r/fdH8/D3TJxVsk12yXcSkQNySE5Jh3QJJ3fknvwnD8G34FdwEvx5Tg1qs5qvZM6CzhPD2M7+</latexit> L <latexit sha1_base64="03loGMSM0SK+KtU6mj3Y2dowETQ=">AAACgHicdVHLSsNAFJ3GV62vVpduBkuhq5qooLgqunHhokJfkIYymUzaoTOZMDMRS+hnuNXv8m+ctFk0rV64cDj3dTjXjxlV2rZ/StbO7t7+QfmwcnR8cnpWrZ33lUgkJj0smJBDHynCaER6mmpGhrEkiPuMDPzZc1YfvBOpqIi6eh4Tj6NJREOKkTaUO+JITzFi6etiXK3bLXsZcBs4OaiDPDrjWmk8CgROOIk0Zkgp17Fj7aVIaooZWVRGiSIxwjM0Ia6BEeJEeelS8wI2DBPAUEiTkYZLdn0iRVypOfdNZ6ZRbdYy8q+am+jwwUtpFCeaRHh1KEwY1AJmBsCASoI1mxuAsKRGK8RTJBHWxqbCpq7jpZm4bE3hvM8XlQZcZzIZseYfxT7EJsJcmPJ/aIq3B2JFEuOqCIyB5iXO5gO2Qf+m5dy2bt7u6u2n/DllcAmuQBM44B60wQvogB7AQIBP8AW+LctqWteWs2q1SvnMBSiE9fgL6q/GbA==</latexit> see also Cremer et al., 2018 Figure 3.1: The Amortization Gap. Optimization surface of L (in nats) for a 2-D latent Gaussian model and an MNIST data example. Shown on the plots are the optimal estimate (F), the output of a direct inferenceMarino, model (), and2018 an 41 Yue, Mandt, optimization trajectory of gradient ascent (•), initialized at the blue square at (0,0). The plot on the right shows an enlarged view near the optimum. Gradient-based optimization outperforms the direct inference model, exhibiting an amortization gap in performance: L (F) > L (•) > L (). through the iterative estimation procedure, ultimately arrives at a better final estimate. The inability of inference models to reach optimal approximate posterior estimates, as typically compared with gradient-based methods, creates an amortization gap (Krishnan, Liang, and Hoffman, 2018; Cremer, Li, and Duvenaud, 2018), which impairs model performance. This is because, within the same parametric distribution family, a sub-optimal amortized approximate posterior, 𝑞 𝜙 , creates a looser ELBO than the optimal approximate posterior, denoted 𝑞 ∗ : log 𝑝 𝜃 (x) ≥ L (x; 𝑞 ∗ , 𝜃) ≥ L (x; 𝑞 𝜙 , 𝜃). The difference between L (x; 𝑞 ∗ , 𝜃) and L (x; 𝑞 𝜙 , 𝜃) is precisely the amortization gap (Cremer, Li, and Duvenaud, 2018). Additional latent dimensions and more complex data or models could further exacerbate this gap. Lack of Prior Information An altogether distinct issue with direct inference models concerns their lack of prior information. Naïve formulations of direct inference models consider a direct mapping from observations, x, to the parameters of the approximate posterior (Eq. 2.28). With a constant prior, e.g., 𝑝 𝜃 (z) = N (z; 0, I) in conventional VAEs (Kingma and Welling, 2014), such a direct mapping is sufficient to estimate the optimal approximate posterior within the class of diagonal distributions. However, in structured models, 41 Figure 3.2: Lack of Prior Information. Naïve direct inference models, encoding only observations, cannot account for conditional priors in structured latent variable models. These arise in hierarchical (left) and sequential (right) latent variable models. For instance, in hierarchical models, bottom-up approximate posteriors at 36 intermediate levels cannot account for conditional “top-down” priors which result from sampling higher-level latent variables. e.g., hierarchical or autoregressive, the situation is more complicated. This is because the prior is now conditional, dependent on samples from upstream (parent) latent variables. If the inference model is only conditioned on the observation, it does not have access to the previously sampled latent variables and therefore cannot account for the conditional prior. For instance, consider a hierarchical latent variable model with following prior: 1:𝐿 𝑝 𝜃 (z )= 𝐿 Ö 𝑝 𝜃 (zℓ |zℓ+1:𝐿 ). ℓ=1 A naïve direct inference model, i.e., only encoding observations, corresponds to the following approximate posterior factorization: 𝑞 𝜙 (z1:𝐿 |x) = 𝐿 Ö 𝑞 𝜙 (zℓ |x). ℓ=1 Plugging these factorizations into the ELBO, we have   L (x; 𝜃, 𝑞) = E𝑞 𝜙 (z1:𝐿 |x) log 𝑝 𝜃 (x|z1:𝐿 ) − 𝐷 KL (𝑞 𝜙 (z1:𝐿 |x)|| 𝑝 𝜃 (z1:𝐿 )) # " 𝐿 ℓ |x) Õ   𝑞 (z 𝜙 = E𝑞 𝜙 (z1:𝐿 |x) log 𝑝 𝜃 (x|z1:𝐿 ) − E𝑞 𝜙 (zℓ:𝐿 |x) log . ℓ |zℓ+1:𝐿 ) 𝑝 (z 𝜃 ℓ=1 The prior at each level depends on zℓ+1:𝐿 while the approximate posterior does not. Thus, at best, a naïve direct inference model can only model the marginal prior, 42 𝑝 𝜃 (zℓ ), unable to capture the structured latent dependencies. This limitation was hinted at by Dayan et al., 1995, noting that “the top-down generative model plays no direct role” in inference, but that “such effects are important in real perception.” A solution to this issue was proposed by Sønderby et al., 2016 and Salimans, 2016, reusing parts of the generative model during inference. Note that the same issue arises in sequential latent variable models, where naïve direct inference models do not have access to latent prior estimates. A similar solution is used in these situations, often using recurrent networks to condition on past latent variables and observations (Chung et al., 2015). While these solutions work empirically, they require hand-crafting the inference procedure. 3.3 Iterative Amortized Inference Learning to Iteratively Optimize While direct inference networks have provided significant benefits in computational efficiency for performing approximate inference, such models can be inaccurate and require additional considerations in structured models (Section 3.2). Dayan et al., 1995 put forth one possible solution: “using iterative recognition, in which the generative and recognition activations interact to produce the final activity of a unit.” Thus, to improve upon the direct inference model paradigm, we pose the following question: can we retain the computational efficiency of inference models while incorporating more powerful and general iterative estimation capabilities? Our proposed solution is a new class of inference models, capable of learning how to update approximate posterior estimates by encoding gradients or errors. Due to the iterative nature of these models, we refer to them as iterative inference models. Through an analysis with latent Gaussian models, we show that iterative inference models generalize direct inference models (Section 3.4) and naturally are capable of performing structured variational inference (Section 3.4 & Chapter 4). Our approach relates to learning to learn (Andrychowicz et al., 2016), where an optimizer model learns to optimize the parameters of an optimizee model. The optimizer receives the optimizee’s parameter gradients and outputs updates to these parameters to improve the optimizee’s loss. The optimizer itself can be learned due to the differentiable computation graph. Such models can adaptively adjust step sizes, potentially outperforming conventional optimizers. For inference optimization, previous works have combined direct inference models with gradient updates (Hjelm et al., 2016; Krishnan, Liang, and Hoffman, 2018; Kim et al., 2018), however, these works do not learn to iteratively optimize. Putzky and Welling, 2017 use recurrent 43 <latexit sha1_base64="HlSpur9H0Rx2BbfN6EKhfq3sCbs=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxWaEv6Awlk0nb0GQSkoxYhv6GW/0t/8ZMO4s+9ELgcO65uYdzQ8moNq77U3CKe/sHh6Wj8vHJ6dl5pXrR0yJRmHSxYEINQqQJozHpGmoYGUhFEA8Z6Yezl6zffydKUxF3zFySgKNJTMcUI2Mp3w956jMrj9BiVKm5DXdZcBd4OaiBvNqjamHkRwInnMQGM6T10HOlCVKkDMWMLMp+oolEeIYmZGhhjDjRQbo0vYB1y0RwLJR9sYFLdn0iRVzrOQ+tkiMz1du9jPyrN0zM+DFIaSwTQ2K8WjROGDQCZgnAiCqCDZtbgLCi1ivEU6QQNjanjZ86XpBm5rJvNtaHfFGuw3UmsyEN/9jUITYRdsOU/0NTvDsgNUlsqiKyAdqTeNsH2AW9ZsO7azTf7mut5/w4JXAFrsEN8MADaIFX0AZdgIEEn+ALfDtF59ZxneZK6hTymUuwUc7TL+fPxtw=</latexit> ✏ <latexit sha1_base64="mlLz5PRn6k/2I/uXqfF1v+kMzi4=">AAACgnicdVFNS0JBFB1fVmZfWss2QyJEC3kvg1q0kNq0NMgK9CHzxqsOzVcz8yJ5+Dva1s/q3zRPXWTWhQuHc78O5yaaM+vC8KsQrBXXNzZLW+Xtnd29/Ur14MGq1FDoUMWVeUqIBc4kdBxzHJ60ASISDo/J801ef3wFY5mS926iIRZkJNmQUeI8FfcSkfVAW8aVnPYrtbARzgKvgmgBamgR7X610O8NFE0FSEc5sbYbhdrFGTGOUQ7Tci+1oAl9JiPoeiiJABtnM9VTXPfMAA+V8SkdnrE/JzIirJ2IxHcK4sb2dy0n/6p1Uze8jDMmdepA0vmhYcqxUzi3AA+YAer4xANCDfNaMR0TQ6jzRi1tuo/iLBeXr1k6n4hpuY5/MrkM7cTbch/hI+UvjMU/NKOrA9pC6l1VA2+gf0n0+wGr4OGsETUbZ3fntdb14jkldISO0QmK0AVqoVvURh1E0Qt6Rx/oMygGp0EUNOetQWExc4iWIrj6Bk+fx38=</latexit> z <latexit sha1_base64="JJBRHMAw1+FJL17Y9ikHFHSRDGA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuyIyaqDuiG5eY8IowIZ1OBxr6mLQdI074C7f6X/6NHWDBgN6kycm5j3N6bxAzqo3r/hScre2d3b3ifung8Oj4pFw57WiZKEzaWDKpegHShFFB2oYaRnqxIogHjHSDyVOW774RpakULTONic/RSNCIYmQs9TrgyIyDKP2YDctVt+7OA24CbwmqYBnNYaUwHIQSJ5wIgxnSuu+5sfFTpAzFjMxKg0STGOEJGpG+hQJxov10bnkGa5YJYSSVfcLAObvakSKu9ZQHtjKzqNdzGflXrp+Y6N5PqYgTQwReCEUJg0bC7P8wpIpgw6YWIKyo9QrxGCmEjd1SblLL89PMXDYmJx/wWakGV5nMRmz4e74OsZG0CmP+D03xZkOsSWK3KkO7QHsSb/0Am6BzXfdu6tcvt9XG4/I4RXAOLsAV8MAdaIBn0ARtgIEAn+ALfDsF59KpO+6i1Ckse85ALpyHX0lCxig=</latexit> Figure 3.3: Iterative Amortized Inference. Computation graph for a latent variable model with iterative amortized inference. Red dots, denoting errors or gradients from the ELBO objective, are used to update the current estimate of the approximate posterior parameters, λ. Using the pathwise derivative estimator, one can update the inference model parameters, 𝜙, learning to efficiently perform iterative inference. 10 inference models for MAP estimation of denoised images in linear models. We propose a single method for learning to perform variational inference, generally applicable to latent variable models. Our work extends techniques for learning to optimize along several directions, discussed in Section 3.4. Iterative Inference Models Following the notation from Chapter 2, we denote an iterative amortized inference model as 𝑓𝜙 with parameters 𝜙. If we consider the case of a single data example, x, and corresponding approximate posterior distribution parameter estimate, λ, the basic form of an iterative amortized inference model is given as: λ ← 𝑓𝜙 (∇λ L, λ), (3.1) where we have used the shorthand L ≡ L(x; 𝜃, 𝑞). Iterative inference models take in the current estimate of λ, as well as the inference gradient, ∇λ L, and output an updated estimate of λ. As with direct inference models, iterative inference model parameters are updated using estimates of ∇𝜙 L, obtained through the pathwise derivative estimator (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014) or the score function estimator (Gregor, Danihelka, Mnih, et al., 2014; Ranganath, Gerrish, and Blei, 2014). While we consider other forms of iterative inference models in Section 3.4, note that Eq. 3.1 is a generalization of basic stochastic gradient-based optimization. For instance, one special case is λ ← λ + 𝛼∇λ L, 44 however, Eq. 3.1 also contains more general non-linear updates (Andrychowicz et al., 2016). Figure 3.3 displays a simplified computation graph of the inference procedure. We discuss more specific design choices in Section 3.5. 3.4 Iterative Inference in Latent Gaussian Models Latent Gaussian models are used in both VAEs (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014) and hierarchical predictive coding (Rao and Ballard, 1999; Friston, 2005), so we focus on iterative amortized inference in these models. Latent Gaussian models have Gaussian prior densities over latent variables: 𝑝(z) = N (z; µ 𝑝 , diag σ 2𝑝 ).1 While the approximate posterior can be any probability density, it is typically also chosen as Gaussian: 𝑞(z|x) = N (z; µ𝑞 , diag σ𝑞2 ). With   this choice, λ ≡ µ𝑞 , σ𝑞 . We can express Eq. 3.1 for this setup as:   µ𝑞 , σ𝑞 ← 𝑓𝜙 (∇µ𝑞 L, ∇σ𝑞 L, µ𝑞 , σ𝑞 ), (3.2) In Marino, Yue, and Mandt, 2018, we derive the gradients ∇µ𝑞 L and ∇σ𝑞 L for the cases where 𝑝 𝜃 (x|z) takes a Gaussian and Bernoulli form, though any output distribution can be used. Generally, these gradients are composed of 1) errors, expressing the mismatch in distributions, and 2) Jacobian matrices, which invert the generative mappings. For instance, assuming a Gaussian output density, 𝑝(x|z) = N (x; µx , diag σx2 ), the gradient for µ𝑞 is ∇µ𝑞 L = J| εx − εz , (3.3) where the Jacobian, J, and observed and latent errors, εx and εz , are defined as   𝜕µx J ≡ Ez∼𝑞(z|x) , (3.4) 𝜕µ𝑞   x − µx εx ≡ Ez∼𝑞(z|x) , σx2 " # z − µ𝑝 εz ≡ Ez∼𝑞(z|x) . σ 2𝑝 (3.5) (3.6) Here, we have assumed µx is a function of z and σx2 is a global parameter. The gradient ∇σ𝑞 L is composed of similar terms as well as an additional term penalizing approximate posterior entropy. Inspecting and understanding the composition of the 1 Although the prior is typically a standard Normal density, we use this prior form for generality. 45 (a) Direct (b) Iterative Figure 3.4: Automatic Top-Down Inference. Bottom-up direct amortized inference cannot account for conditional priors, whereas iterative amortized inference automatically accounts for these priors through gradients or errors. 37 37 gradients reveals the forces pushing the approximate posterior toward agreement with the data, through εx , and agreement with the prior, through εz . In other words, inference is as much a “top-down” process as it is a “bottom-up” process, and the optimal combination of these terms is given by the approximate posterior gradients. Interpreting Top-Down Inference Naïve direct inference models, which only encode the data, lack the top-down prior information encapsulated in εz . For instance, in a chain-structured hierarchical latent variable model (Figure 3.2, left), the gradient of µℓ𝑞 , the approximate posterior mean at layer ℓ, is ℓ ∇µℓ𝑞 L = Jℓ| εℓ−1 (3.7) z − εz , where Jℓ is the Jacobian of the generative mapping at layer ℓ and εℓz is defined similarly to Eq. 3.6. The error εℓz depends on the top-down prior at layer ℓ, which, unlike the single-level case, varies across data examples. Thus, a purely bottom-up inference procedure will struggle to optimize, and therefore utilize, the conditional prior. Iterative inference models, which rely on approximate posterior gradients, naturally account for both bottom-up and top-down influences (Figure 3.4). Approximating Approximate Posterior Derivatives In the formulation of iterative inference models given in Eq. 3.1, inference optimization is restricted to first-order approximate posterior derivatives. Thus, it may require many inference iterations to reach reasonable approximate posterior estimates. Rather than calculate costly higher-order derivatives, we can take a different approach. 46 Approximate posterior derivatives (e.g., Eq. 3.3 and higher-order derivatives) are essentially defined by the errors at the current estimate, as the other factors, such as the Jacobian matrices, are internal to the model. Thus, the errors provide more general information about the curvature beyond the gradient. As iterative inference models already learn to perform approximate posterior updates, it is natural to ask whether the errors provide a sufficient signal for faster inference optimization. In other words, we may be able to offload approximate posterior derivative calculation onto the inference model, yielding a model that requires fewer inference iterations while maintaining or possibly improving computational efficiency. Comparing with Eq. 3.2, the form of this new iterative inference model is   µ𝑞 , σ𝑞 ← 𝑓𝜙 (εx , εz , µ𝑞 , σ𝑞 ). (3.8) In Section 3.5, we empirically find that models of this form converge to better solutions than gradient-encoding models when given fewer inference iterations. It is also worth noting that this error encoding scheme is similar to DRAW (Gregor, Danihelka, Graves, et al., 2015). However, in addition to architectural differences in the generative model, DRAW and later extensions (Gregor, Besse, et al., 2016) do not include top-down errors nor error weighting. This encoding form is also similar to PredNet (Lotter, Kreiman, and Cox, 2017), which is also inspired by predictive coding, but is not strictly formulated in terms of variational inference. Generalizing Direct Inference Models Under certain assumptions on single-level latent Gaussian models, iterative inference models of the form in Section 3.4 generalize direct inference models. First, note that εx (Eq. 3.5) is an affine transformation of x: εx = Ax + b, (3.9)   A ≡ E𝑞(z|x) (diag σx2 ) −1 , (3.10) where   µx b ≡ −E𝑞(z|x) . (3.11) σx2 Making the reasonable assumption that the initial approximate posterior and prior are both constant, then in expectation, A, b, and εz are constant across all data examples at the initial inference iteration. Using proper weight initialization and input normalization, it is equivalent to input x or an affine transformation of x into a 47 Figure 3.5: Iterative Amortized Inference Optimization. Optimization trajectory on L (in nats) for an iterative inference model with a 2D latent Gaussian model for a particular MNIST example. The estimate is initialized at (0, 0) (cyan square), and the iterative inference model adaptively adjusts inference update step sizes to 34 estimate () to arrive at the optimum (F). iteratively refine the approximate posterior fully-connected neural network. Therefore, in this case, direct inference models are equivalent to the special case of a one-step iterative inference model. Thus, we can interpret standard inference models as learning a map of local curvature around a fixed approximate posterior estimate. Iterative inference models, in contrast, learn to traverse the optimization landscape more generally. 3.5 Experiments Setup Data Using latent Gaussian models, we empirically evaluate iterative amortized inference on image and text data. For images, we use MNIST (LeCun et al., 1998), Omniglot (Lake, Salakhutdinov, and Tenenbaum, 2013), Street View House Numbers (SVHN) (Netzer et al., 2011), and CIFAR-10 (Krizhevsky and Hinton, 2009). MNIST and Omniglot are dynamically binarized and modeled with Bernoulli conditional likelihoods, and SVHN and CIFAR-10 are modeled with Gaussian conditional likelihoods, using the procedure from (Gregor, Besse, et al., 2016). For text, we use RCV1 (Lewis et al., 2004), with word count data modeled with a multinomial output. Models Unlike direct inference models, which deal with a fixed input domain, approximate posterior parameters, errors, and inference gradients change throughout inference and learning. We found it beneficial to separately normalize each input 48 using layer normalization (Ba, Kiros, and Hinton, 2016). We also found it useful, though not necessary, to input the data observation (Figure 3.9a). For comparison with direct amortization, all experiments use feedforward networks, though recurrent networks may yield improved performance (Andrychowicz et al., 2016). For the output, we use a gated (“highway” (Srivastava, Greff, and Schmidhuber, 2015)) update: λ ← g𝜙 (·) λ + (1 − g𝜙 (·)) δ𝜙 (·), (3.12) where g𝜙 (·) ∈ [0, 1] is the gate and δ𝜙 (·) is the update, both of which are output by the iterative inference model, and denotes element-wise multiplication. We estimate ∇𝜙 L using the pathwise derivative estimator, averaging over inference iterations. Reported values of L are estimated using 1 latent sample, and reported values of log 𝑝(x) and perplexity (Tables 3.1, 3.2, & 3.3) are estimated using 5, 000 importance-weighted samples. Additional experiment details can be found in Marino, Yue, and Mandt, 2018, and accompanying code can be found at github.com/joelouismarino/iterative_inference. Approximate Inference Optimization We start by demonstrating the inference optimization capabilities of iterative inference models. These models indeed learn to perform inference optimization through an adaptive iterative estimation procedure, with the results highlighting the unique aspects of iterative amortization as compared with direct amortization. We show that iterative inference models utilize multiple inference iterations rather than collapsing to static, one-step encoders. 2D Visualization As in Section 3.2, we directly visualize iterative amortized inference optimization in a 2D latent Gaussian model trained on MNIST with a point estimate approximate posterior. Model architectures are identical to those used in Section 3.2, with additional details found in Marino, Yue, and Mandt, 2018. Figure 3.5 shows a 16-step inference optimization trajectory taken by the iterative inference model for a particular example. The model adaptively adjusts inference update step sizes to navigate the optimization surface, quickly arriving and remaining at a near-optimal estimate. This is in contrast of other iterative inference optimization schemes, e.g., Krishnan, Liang, and Hoffman, 2018 and Kim et al., 2018, which must tune the step size for gradient-based optimizers. 49 (a) Iterations (b) Wall Clock Time Figure 3.6: Amortized vs. Gradient-Based Optimization. Average ELBO over MNIST validation set during inference optimization as a function of (a) inference iterations and (b) inference wall-clock time. Iterative amortized inference converges faster than gradient-based optimizers to better estimates, remaining stable over hundreds of iterations, despite only being trained with 16 inference iterations. ELBO During Inference To compare iterative amortization with gradient-based optimizers, we can quantify and compare optimization performance through the ELBO. In Figure 3.6, we plot the average ELBO on the MNIST validation set during inference for each optimizer, comparing in terms of iterations (Figure 3.6a) and wall clock time (Figure 3.6b). On average, iterative amortized inference converges significantly faster to better estimates than gradient-based optimizers. Note that, in this case, the inference model has less information than the optimizers; it only has access to the local gradient, whereas the optimizers use momentum and similar terms. Also note that the model’s final estimates are empirically stable over 100s of iterations, despite only being trained using 16 inference iterations. Reconstructions Iterative amortized inference optimization can also be visualized through image reconstructions. We demonstrate this for each dataset in Figure 3.7. As the reconstruction term is typically the dominant term in L, the output reconstructions improve in terms of visual quality during inference optimization (left to right), more closely resembling x (far right of each figure). Gradient Magnitudes During inference optimization, iterative inference models attempt to find local maxima. The gradient magnitudes of the ELBO w.r.t. the approximate posterior parameters, i.e., ∇λ L, should thus decrease during inference. In Figure 3.8, we plot these gradient magnitudes over inference iterations (each curve) throughout training for a model trained on RCV1. We find that the gradient magnitudes do, indeed, decrease during inference, with diminishing reduction at 50 Figure 3.7: Reconstructions Over Inference Iterations. During inference (left to right), reconstruction means become gradually sharper, more closely resembling data examples (right). Figure 3.8: Reduced Gradient Magnitudes. Gradient magnitudes (vertical axis) over inference iterations (indexed by color on right) during training (horizontal axis) on RCV1. Approximate posterior mean gradient magnitudes decrease over inference iterations as estimates approach local maxima. later iterations. Additional Inference Iterations & Latent Samples We highlight two sources that enable iterative inference models to further improve performance: additional 1) inference iterations and 2) latent samples. Additional inference iterations allow the inference model to further refine approximate posterior estimates. We demonstrate this on binarized MNIST, using inference models that encode approximate posterior gradients (∇λ L) or errors (εx , εz ), with or without the data (x). These inference models are trained with 2, 5, 10, and 16 inference iterations. The model architecture is identical in each case, the only difference being the number 51 (a) (b) Figure 3.9: Hyperparameter Comparison. ELBO for direct and iterative inference models on binarized MNIST for (a) additional inference iterations during training and (b) additional samples. Iterative inference models improve significantly with both quantities. of input weights. Note that the small size of z relative to x gives the gradient encoding model (∇λ ) fewer input parameters than the direct inference model (solid black line). The other inference models have more input parameters. Results are shown in Figure 3.9a, where we observe improved performance with increasing inference iterations. With enough iterations, each iterative inference model outperforms the direct inference model. As discussed in Section 3.4, encoding errors approximates higher-order derivatives, which helps when training with fewer inference iterations. Additional approximate posterior samples provide more precise (lower variance) gradient and error estimates, potentially allowing an iterative inference model to output improved updates. To verify this, we train direct and iterative inference models on binarized MNIST using 1, 5, 10, and 20 approximate posterior samples. Here, iterative inference models encode the data, x, and approximate posterior gradients, ∇λ L, for 5 iterations. Results are shown in Figure 3.9b. Iterative inference models improve by more than 1 nat with additional samples, further widening the improvement over the comparable direct inference model. Comparison with Direct Amortized Inference We now compare the resulting log-likelihood performance between direct and iterative amortized inference on binarized MNIST, CIFAR-10, and RCV1. Inference model architectures are identical across each comparison, again, with the exception of input parameters. Further details are found in Marino, Yue, and Mandt, 2018. Tables 3.1 & 3.2 report estimated negative log-likelihood performance on binarized MNIST and CIFAR-10 respectively. Table 3.3 reports estimated perplexity on RCV1. 52 Table 3.1: Negative Log-Likelihood on binarized MNIST (in nats) for direct and iterative amortized inference. Table 3.2: Negative Log-Likelihood on CIFAR-10 (in bits/dim.) for direct and iterative amortized inference. − log 𝑝(x) Single-Level Direct Iterative Hierarchical Direct Iterative − log 𝑝(x) Single-Level Direct Iterative Hierarchical Direct Iterative 84.14 ± 0.02 83.84 ± 0.05 82.63 ± 0.01 82.457 ± 0.001 5.823 ± 0.001 5.64 ± 0.03 5.565 ± 0.002 5.456 ± 0.005 Table 3.3: Perplexity on RCV1 for direct and iterative amortized inference. ≤ Krishnan, Liang, and Hoffman (2018) 331 Direct 323 ± 3 377.4 ± 0.5 Iterative 285.0 ± 0.1 314 ± 1 Perplexity Briefly, Perplexity, 𝑃, is defined as 𝑃 ≡ exp − 1 Õ 1 𝑁 𝑖 ! log 𝑝(x (𝑖) ) , 𝑛𝑖 where 𝑁 is the number of data examples and 𝑛𝑖 is the total number of word counts in document 𝑖. On each dataset and model, iterative amortized inference outperforms direct amortization. This holds for both single-level and hierarchical (2 level) models. We observe larger improvements on the high-dimensional RCV1 dataset, consistent with Krishnan, Liang, and Hoffman, 2018. Because the generative model architectures are kept fixed, performance improvements demonstrate improvements in the inference procedure. 3.6 Discussion In this chapter, we introduced iterative amortized inference, which learns to refine inference estimates by encoding approximate posterior gradients or errors, i.e., learned negative feedback perception. This inference procedure is inspired by hierarchical predictive coding (Rao and Ballard, 1999; Friston, 2005), which uses Gaussian errors to perform gradient-based variational inference. On several benchmark datasets of images and text, we have demonstrated that iterative amortized inference empirically outperforms direct amortization, while retaining the efficiency benefits of amortization over gradient-based optimization. In latent Gaussian models, iterative 53 amortized inference generalizes and extends direct amortization, and by naturally accounting for priors through gradients or errors, these models provide insight and justification for top-down inference. This chapter has focused exclusively on performing inference in static latent variable models. In the following chapter, we apply this inference technique to sequential latent variable models, where we devise a general-purpose filtering procedure centered around prediction and updating. References Andrychowicz, Marcin et al. (2016). “Learning to learn by gradient descent by gradient descent”. In: Advances in Neural Information Processing Systems (NIPS), pp. 3981–3989. Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E Hinton (2016). “Layer normalization”. In: arXiv preprint arXiv:1607.06450. Chung, Junyoung et al. (2015). “A recurrent latent variable model for sequential data”. In: Advances in neural information processing systems, pp. 2980–2988. Cremer, Chris, Xuechen Li, and David Duvenaud (2018). “Inference Suboptimality in Variational Autoencoders”. In: International Conference on Machine Learning, pp. 1078–1086. Dayan, Peter et al. (1995). “The helmholtz machine”. In: Neural computation 7.5, pp. 889–904. Friston, Karl (2005). “A theory of cortical responses”. In: Philosophical Transactions of the Royal Society of London B: Biological Sciences 360.1456, pp. 815–836. Gregor, Karol, Frederic Besse, et al. (2016). “Towards conceptual compression”. In: Advances In Neural Information Processing Systems (NIPS), pp. 3549–3557. Gregor, Karol, Ivo Danihelka, Alex Graves, et al. (2015). “DRAW: A recurrent neural network for image generation”. In: Proceedings of the International Conference on Machine Learning (ICML), pp. 1462–1471. Gregor, Karol, Ivo Danihelka, Andriy Mnih, et al. (2014). “Deep autoregressive networks”. In: Proceedings of the International Conference on Machine Learning (ICML), pp. 1242–1250. Hjelm, Devon et al. (2016). “Iterative refinement of the approximate posterior for directed belief networks”. In: Advances in Neural Information Processing Systems (NIPS), pp. 4691–4699. Kalman, Rudolph Emil (1960). “A new approach to linear filtering and prediction problems”. In: Journal of Basic Engineering 82.1, pp. 35–45. Kim, Yoon et al. (2018). “Semi-Amortized Variational Autoencoders”. In: Proceedings of the International Conference on Machine Learning (ICML). 54 Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Krishnan, Rahul G, Dawen Liang, and Matthew Hoffman (2018). “On the challenges of learning with inference networks on sparse, high-dimensional data”. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 143–151. Krizhevsky, Alex and Geoffrey E Hinton (2009). “Learning multiple layers of features from tiny images”. In: Lake, Brenden M, Ruslan R Salakhutdinov, and Josh Tenenbaum (2013). “One-shot learning by inverting a compositional causal process”. In: Advances in Neural Information Processing Systems (NIPS), pp. 2526–2534. LeCun, Yann et al. (1998). “Gradient-based learning applied to document recognition”. In: Proceedings of the IEEE 86.11, pp. 2278–2324. Lewis, David D et al. (2004). “Rcv1: A new benchmark collection for text categorization research”. In: The Journal of Machine Learning Research 5.Apr, pp. 361–397. Lotter, William, Gabriel Kreiman, and David Cox (2017). “Deep predictive coding networks for video prediction and unsupervised learning”. In: International Conference on Learning Representations. Marino, Joseph, Yisong Yue, and Stephan Mandt (2018). “Iterative Amortized Inference”. In: International Conference on Machine Learning, pp. 3403–3412. url: http://proceedings.mlr.press/v80/marino18a.html. Mumford, David (1992). “On the computational architecture of the neocortex”. In: Biological cybernetics 66.3, pp. 241–251. Netzer, Yuval et al. (2011). “Reading digits in natural images with unsupervised feature learning”. In: NIPS workshop on deep learning and unsupervised feature learning. Putzky, Patrick and Max Welling (2017). “Recurrent inference machines for solving inverse problems”. In: arXiv preprint arXiv:1706.04008. Ranganath, Rajesh, Sean Gerrish, and David Blei (2014). “Black box variational inference”. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 814–822. Rao, Rajesh PN and Dana H Ballard (1999). “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” In: Nature neuroscience 2.1. 55 Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Salimans, Tim (2016). “A structured variational auto-encoder for learning deep hierarchies of sparse features”. In: arXiv preprint arXiv:1602.08734. Sønderby, Casper Kaae et al. (2016). “Ladder variational autoencoders”. In: Advances in Neural Information Processing Systems (NIPS), pp. 3738–3746. Srivastava, Rupesh K, Klaus Greff, and Jürgen Schmidhuber (2015). “Training very deep networks”. In: Advances in neural information processing systems (NIPS), pp. 2377–2385. 56 Chapter 4 AMORTIZED VARIATIONAL FILTERING learned negative feedback (dynamic) perception Marino, Joseph, Milan Cvitkovic, and Yisong Yue (2018). “A general method for 10 amortizing variational filtering”. In: Advances in Neural Information Processing Systems, pp. 7857–7868. url: http://papers.nips.cc/paper/8011- ageneral-method-for-amortizing-variational-filtering. 4.1 Introduction Chapter 3 introduced iterative amortized inference, a technique for accurately and efficiently performing inference optimization using gradients or errors. While inference in this setup is iterative, the models themselves were formulated in static settings, e.g., images. Many data sources, in contrast, contain sequential structure, e.g., video, audio, or proprioceptive kinematics. This structure can be exploited to improve modeling performance and enable prediction of future observations. Thus, we require techniques for formulating deep sequential latent variable models, as well as inference techniques that enable their training. In this chapter, we extend iterative amortized inference to perform filtering in deep sequential latent variable models, providing a general-purpose algorithm for performing approximate Bayesian filtering. We first formulate variational EM in the filtering setting, i.e., conditioning only on past and current variables. We then describe an instantiation of this algorithm, implementing inference optimization at each step with iterative amortized inference. The resulting inference method, amortized variational filtering (AVF), is generally applicable to sequential latent 57 variable models, while retaining the efficiency of amortization. We demonstrate this generality by applying AVF to three previously proposed deep sequential latent variable models across three data domains: audio, video, and MIDI music. In comparison with the custom inference models proposed for each generative model, we show that our single method, AVF, matches or outperforms log-likelihood performance in each case, with further improvement resulting from additional inference iterations. Thus, AVF provides a conceptually simple, generally applicable, and powerful filtering inference procedure. 4.2 Background Sequential Latent Variable Models Here, we briefly review sequential latent variable models (SLVMs). A sequence of 𝑇 observations, x1:𝑇 , can be modeled using a SLVM, 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ), which models the joint distribution between x1:𝑇 and a sequence of latent variables, z1:𝑇 , with parameters 𝜃. It is typically assumed that 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) can be factorized into conditional joint distributions at each step, 𝑝 𝜃 (x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ), which are conditioned on preceding variables. This results in the following autoregressive form: 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) = 𝑇 Ö 𝑝 𝜃 (x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) = 𝑡=1 𝑇 Ö 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). (4.1) 𝑡=1 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) is the observation model, and 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ) is the latent dynamics model, both of which can be arbitrary functions of their conditioning variables. However, while Eq. 4.1 provides the general form of a SLVM, further assumptions about the dependency structure, e.g., Markov, or functional forms, e.g., linear, are often necessary for tractable inference and learning. Variational Inference Given a model and a set of observations, we typically want to infer the posterior for each sequence, 𝑝 𝜃 (z1:𝑇 |x1:𝑇 ), and learn the model parameters, 𝜃. Inference can be performed online or offline through Bayesian filtering or smoothing respectively (Särkkä, 2013), and learning can be performed through maximum likelihood estimation. Unfortunately, inference and learning are intractable for all but the simplest, e.g., linear, model classes. For non-linear functions, which are present in deep latent variable models, we must resort to approximate inference. As introduced in Chapter 2, variational inference (Jordan et al., 1998), framed in the sequential setting, reformulates inference as optimization by introducing an approximate posterior, 𝑞(z1:𝑇 |x1:𝑇 ), then minimizing the KL divergence to the true posterior, 𝑝 𝜃 (z1:𝑇 |x1:𝑇 ). 58 To avoid evaluating 𝑝 𝜃 (z1:𝑇 |x1:𝑇 ), one can express the KL divergence as 𝐷 KL (𝑞(z1:𝑇 |x1:𝑇 )|| 𝑝 𝜃 (z1:𝑇 |x1:𝑇 )) = log 𝑝 𝜃 (x1:𝑇 ) − L (x1:𝑇 ; 𝜃, 𝑞), where, again, L is the evidence lower bound (ELBO), defined as   𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) L (x1:𝑇 ; 𝜃, 𝑞) ≡ E𝑞(z1:𝑇 |x1:𝑇 ) log . 𝑞(z1:𝑇 |x1:𝑇 ) (4.2) (4.3) In Eq. 4.2, log 𝑝 𝜃 (x1:𝑇 ) is independent of 𝑞(z1:𝑇 |x1:𝑇 ), so one can minimize the KL divergence to the true posterior, thereby performing approximate inference, by maximizing L w.r.t. 𝑞(z1:𝑇 |x1:𝑇 ). Further, as KL divergence is non-negative, Eq. 4.2 implies that the ELBO lower bounds the log-likelihood. Therefore, upon maximizing L w.r.t. 𝑞(z1:𝑇 |x1:𝑇 ), one can use the gradient ∇𝜃 L to learn the model parameters. These two optimization procedures are respectively the expectation and maximization steps of the variational EM algorithm (Neal and Hinton, 1998), which alternate until convergence. To scale this algorithm, stochastic gradients can be used for both inference (Ranganath, Gerrish, and Blei, 2014) and learning (Hoffman et al., 2013). As we have noted, performing inference using gradient-based optimization can be computationally inefficient, potentially requiring many inference iterations. To increase efficiency, we can use amortization (Gershman and Goodman, 2014). However, direct inference models (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014), only receiving the data as input, are unable to account for conditional (empirical) priors, which occur with structured latent variables. Such priors arise in the latent dynamics of SLVMs, forming priors across time steps. As with “top-down” inference in hierarchical latent variable models (C. K. Sønderby et al., 2016), many previous works use recurrent inference models in SLVMs, e.g., Chung, Kastner, et al., 2015, to encode previous observations and latent samples. However, in this case, the inference model must effectively re-learn the latent dynamics, possibly introducing inaccuracies into the inference scheme. Related Work Many deterministic deep sequential models have been proposed for sequence data (Chung, Gulcehre, et al., 2014; N. Srivastava, Mansimov, and Salakhudinov, 2015; Lotter, Kreiman, and Cox, 2017; Finn, Goodfellow, and Levine, 2016). While these models often capture many aspects of the data, they cannot account for the uncertainty inherent in many domains, typically arising from partial observability of the environment. By averaging over multi-modal distributions, these models 59 often produce samples in regions of low probability, e.g., blurry video frames. This inadequacy necessitates moving to models with stochastic latent variables, which can better model uncertainty to accurately capture the distribution of possible sequences. Amortized variational inference (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014) has enabled many recently proposed probabilistic deep sequential latent variable models, with applications to video (Walker et al., 2016; Karl et al., 2017; Xue et al., 2016; Johnson et al., 2016; Gemici et al., 2017; Fraccaro, Kamronn, et al., 2017; Babaeizadeh et al., 2018; Denton and Fergus, 2018; Li and Mandt, 2018; He et al., 2018), speech (Chung, Kastner, et al., 2015; Fraccaro, S. K. Sønderby, et al., 2016; Goyal et al., 2017; Hsu, Zhang, and Glass, 2017; Li and Mandt, 2018), handwriting (Chung, Kastner, et al., 2015), music (Fraccaro, S. K. Sønderby, et al., 2016), etc. While these models differ in their functional mappings, most fall within the general form of Eq. 4.1. Crucially, simply encoding the observation at each step is insufficient to accurately perform approximate inference, as the prior can vary across steps. Thus, with each model, a hand-crafted amortized inference procedure has been proposed. For instance, many filtering inference methods re-use various components of the generative model (Chung, Kastner, et al., 2015; Fraccaro, S. K. Sønderby, et al., 2016; Gemici et al., 2017; Denton and Fergus, 2018), while some methods introduce separate recurrent neural networks into the filtering procedure (Bayer and Osendorfer, 2014; Denton and Fergus, 2018) or encode the previous latent sample (Karl et al., 2017). Specifying an amortized filtering method has been an engineering effort, as we have lacked a general-purpose method. The variational filtering EM algorithm (Section 4.3) precisely specifies the inference optimization procedure implied by performing filtering inference on the sequential variational objective (Eq. 4.3). The main insight from this analysis is that, having drawn approximate posterior samples at previous steps, inference becomes a temporally localized optimization, depending only on the current prior and observation. This suggests a general approach that explicitly performs inference optimization at each step, replacing the current collection of custom filtering methods. When the approximate posterior at each step is initialized at the corresponding prior, this approach entails a Bayesian prediction-update loop, with the update composed of a gradient (or error) signal. Perhaps the closest technique in the probabilistic modeling literature is the “residual” inference method from Fraccaro, S. K. Sønderby, et al., 2016, which updates the approximate posterior mean from the prior. Similar ideas have been proposed 60 on an empirical basis for deterministic models (Lotter, Kreiman, and Cox, 2017; Henaff, Zhao, and LeCun, 2017). PredNet (Lotter, Kreiman, and Cox, 2017) is a deterministic model that encodes prediction errors to perform inference. Like our method, this approach is inspired by hierarchical predictive coding (Rao and Ballard, 1999; Friston, 2005). Predictive coding, in turn, is an approximation to classical Bayesian filtering (Särkkä, 2013), which updates the posterior from the prior using the log-likelihood (error) of the prediction. For linear Gaussian models, this manifests as the Kalman filter (Kalman, 1960), which uses prediction errors to perform exact inference. Finally, several recent works have used particle filtering in conjunction with amortized inference to provide a tighter bound on the log-likelihood for sequential models (Maddison et al., 2017; Naesseth et al., 2018; Le et al., 2018). The techniques developed here can also be applied to this setting, providing an improved method for estimating the proposal distribution. 4.3 Variational Filtering Variational Filtering Expectation Maximization (EM) In the filtering setting, the approximate posterior at each step is conditioned only on information from past and present variables, enabling online approximate inference. This implies a structured approximate posterior, in which 𝑞(z1:𝑇 |x1:𝑇 ) factorizes across steps as 𝑇 Ö 𝑞(z1:𝑇 |x1:𝑇 ) = 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ). (4.4) 𝑡=1 Note that the conditioning variables in each term of 𝑞 denote an indirect dependence that arises through inference optimization and does not necessarily constitute a direct functional mapping. Under a filtering approximate posterior, the ELBO can be expressed as 𝑇 𝑇 Õ Õ e𝑡 , L= EÎ𝑡−1 𝑞(z 𝜏 |x ≤𝜏 ,z<𝜏 ) [L𝑡 ] = L (4.5) 𝜏=1 𝑡=1 𝑡=1 (derived in Appendix 4.6) where L𝑡 is the step ELBO, defined as   𝑝 𝜃 (x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) L𝑡 ≡ E𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) log , 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) (4.6) e𝑡 as the 𝑡 th term in the summation. Note that with a and we have also defined L single step, the filtering ELBO reduces to the first step ELBO, thereby recovering the case of a static latent variable model. As in this setting, the step ELBO can be 61 I t 1 er nf en ce O pt im t i za n io ( ) ep st E t t+1 Figure 4.1: Variational Filtering Inference. The diagram shows filtering inference within a sequential latent variable model, as outlined in Algorithm 2. The central gray region depicts inference optimization at step 𝑡, initialized at or near the corresponding prior (white), 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). Sampling from the approximate posterior (blue) generates the conditional likelihood (green), 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ), which is evaluated at the observation (gray), x𝑡 , to calculate the prediction error. This term is combined with the KL divergence between the approximate posterior and prior, yielding the step ELBO (red), L𝑡 (Eq. 4.7). Inference optimization (E-step) involves finding the approximate posterior that maximizes the step ELBO terms. re-expressed as a reconstruction term and a KL divergence term: L𝑡 = E𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) [log 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 )] − 𝐷 KL (𝑞(z𝑡 |x≤𝑡 , z<𝑡 )|| 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 )). (4.7) The filtering ELBO in Eq. 4.5 is the sum of these step ELBO terms, each of which is evaluated according to expectations over past latent sequences. To perform filtering variational inference, we must find the set of 𝑇 terms in 𝑞(z1:𝑇 |x1:𝑇 ) that maximize the filtering ELBO summation. We now describe the variational filtering EM algorithm, given in Algorithm 2 and depicted in Figure 4.1, which optimizes Eq. 4.5. This algorithm sequentially optimizes each of the approximate posterior terms to perform filtering inference. Consider the approximate posterior at step 𝑡, 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ). This term appears in L, 62 either directly or in expectations, in terms 𝑡 through 𝑇 of the summation: terms in which 𝑞(z𝑡 |x ≤𝑡 , z<𝑡 ) appears z }| { e e e e e e e L = L1 + L2 + · · · + L𝑡−1 + L𝑡 + L𝑡+1 + · · · + L𝑇−1 + L𝑇 . | {z } (4.8) steps on which 𝑞(z𝑡 |x ≤𝑡 , z<𝑡 ) depends However, the filtering setting dictates that the optimization of the approximate posterior at each step can only condition on past and present variables, i.e., steps 1 through 𝑡. Therefore, of the 𝑇 terms in L, the only term through which we can optimize 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) is the 𝑡 th term: e𝑡 . 𝑞 ∗ (z𝑡 |x≤𝑡 , z<𝑡 ) = arg max L (4.9) 𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) e𝑡 requires evaluating expectations over previous approximate posteriors. Optimizing L Again, because approximate posterior estimates cannot be influenced by future variables, these past expectations remain fixed through the future. Thus, variational filtering (the variational E-step) can be performed by sequentially maximizing each L𝑡 w.r.t. 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ), holding the expectations over past variables fixed. Conveniently, once the past expectations have been evaluated, inference optimization is entirely defined by the ELBO at that step. For simple models, such as linear Gaussian models, these expectations may be computed exactly. However, in general, the expectations must be estimated through Monte Carlo samples from 𝑞, with inference optimization carried out using stochastic gradients (Ranganath, Gerrish, and Blei, 2014). As in the static setting, we can initialize 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) at (or near) the prior, 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). This yields a simple interpretation: starting with 𝑞 at the prior, we generate a prediction of the data through the likelihood, 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ), to evaluate the current step ELBO. Using the approximate posterior gradient, we then perform an inference update to the estimate of 𝑞. This resembles classical Bayesian filtering, where the posterior is updated from the prior prediction according to the likelihood of observations. Unlike the classical setting, reconstruction and update steps are repeated until inference convergence. This resembles the extended Kalman filter (Kalman, 1960; Murphy, 2012), but is applicable to non-linear models with non-Gaussian distributions. After inferring an optimal approximate posterior, learning (the variational M-step) can be performed by maximizing the total filtering ELBO w.r.t. the model parameters, 𝜃. As Eq. 4.5 is a summation and differentiation is a linear operation, ∇𝜃 L is the 63 Algorithm 2 Variational Filtering Expectation Maximization 1: Input: observation sequence x1:𝑇 , model 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) 2: ∇𝜃 L = 0 ⊲ parameter gradient 3: for 𝑡 = 1 to 𝑇 do 4: initialize 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) ⊲ at/near 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ) e𝑡 := E𝑞(z |x ,z ) [L𝑡 ] 5: L <𝑡 <𝑡 <𝑡−1 e𝑡 6: 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) = arg max𝑞 L ⊲ inference (E-Step) e𝑡 7: ∇𝜃 L ← ∇𝜃 L + ∇𝜃 L 8: end for 9: 𝜃 = 𝜃 + 𝛼∇𝜃 L ⊲ learning (M-Step) sum of contributions from each of these terms: 𝑇 h i Õ Î ∇𝜃 E 𝑡−1 𝑞(z 𝜏 |x ≤𝜏 ,z<𝜏 ) [L𝑡 ] . ∇𝜃 L = (4.10) 𝜏=1 𝑡=1 Parameter gradients can be estimated online by accumulating the result from each term in the filtering ELBO. The parameters are then updated at the end of the sequence. For large datasets, stochastic estimates of parameter gradients can be obtained from a mini-batch of data examples (Hoffman et al., 2013). Amortized Variational Filtering Performing approximate inference optimization (Algorithm 2, Line 6) with traditional techniques can be computationally costly, requiring many iterations of gradient updates and hand-tuning of optimizer hyper-parameters. In online settings, with large models and data sets, this may be impractical. An alternative approach is to employ an amortized inference model, which can learn to maximize L𝑡 w.r.t. 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) more efficiently at each step. Note that L𝑡 (Eq. 4.6) contains 𝑝 𝜃 (x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) = 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). The prior, 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ), varies across steps, constituting the latent dynamics. Direct inference models, which only encode x𝑡 , do not have access to the prior and therefore cannot properly optimize 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ). Many inference models in the sequential setting attempt to account for this information by including hidden states (Chung, Kastner, et al., 2015; Fraccaro, S. K. Sønderby, et al., 2016; Denton and Fergus, 2018). However, given the complexities of many generative models, it can be difficult to determine how to properly route the necessary prior information into the inference model. As a result, each SLVM has been proposed with an accompanying custom inference model. We propose a simple and general alternative method for amortizing filtering inference that is agnostic to the particular form of the generative model. Iterative 64 amortized inference (Chapter 3) naturally accounts for the changing prior through the approximate posterior gradients or errors. These models are thus a natural candidate for performing inference at each step. Similar to Eq. 3.1, when 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) is a 𝑞 parametric distribution with parameters λ𝑡 , the inference update takes the form: 𝑞 𝑞 e𝑡 ). λ𝑡 ← 𝑓𝜙 (λ𝑡 , ∇λ𝑡𝑞 L (4.11) We refer to this setup as amortized variational filtering (AVF). Again, Eq. 4.11 offers just one particular encoding form for an iterative inference model. For instance, x𝑡 could be additionally encoded at each step. As noted in Chapter 3, in latent Gaussian models, precision-weighted errors provide an alternative inference optimization signal. There are two main benefits to using iterative inference models in the filtering setting: • The approximate posterior is updated from the prior, so model capacity is utilized for inference corrections rather than re-estimating the latent dynamics at each step. • These inference models contain all of the terms necessary to perform inference optimization, providing a simple model form that does not require any additional hidden states or inputs. In practice, these advantages permit the use of relatively simple iterative inference models that can perform filtering inference efficiently and accurately. We demonstrate this in the following section. 4.4 Experiments We empirically evaluate amortized variational filtering using multiple deep sequential latent Gaussian model architectures on a variety of sequence datasets. Specifically, we use AVF to train VRNN (Chung, Kastner, et al., 2015), SRNN (Fraccaro, S. K. Sønderby, et al., 2016), and SVG (Denton and Fergus, 2018) on speech (Garofolo et al., 1993), music (Boulanger-Lewandowski, Bengio, and Vincent, 2012), and video (Schuldt, Laptev, and Caputo, 2004) data. In each setting, we compare AVF against the originally proposed filtering method for the model. Diagrams of the filtering methods are shown in Figure 4.2. Implementations of the models are based on code provided by the respective au- VRNN SRNN inference inference SVG inference AVF 65 zt 1 <latexit sha1_base64="o6yrtT8KWAx2ZOmu7tl4jWX1C8g=">AAACh3icdVFNTwIxEC3rF+DXokcvjcTEE+6qUY6oF4+YiJrAuul2CzS227WdJZIN/8Sr/if/jV3gIKKTNHl5M6/z8iZKBTfgeV8lZ2V1bX2jXKlubm3v7Lq1vQejMk1Zhyqh9FNEDBM8YR3gINhTqhmRkWCP0ctN0X8cMW24Su5hnLJAkkHC+5wSsFTour1I5j1hBTGZhPD8Grp1r+FNCy8Dfw7qaF7tsFYKe7GimWQJUEGM6fpeCkFONHAq2KTaywxLCX0hA9a1MCGSmSCfWp/gI8vEuK+0fQngKftTkRNpzFhGdlISGJrfvYL8q9fNoN8Mcp6kGbCEzhb1M4FB4SIHHHPNKIixBYRqbr1iOiSaULBpLfx07wd5Ya74ZmF9JCfVI/yTKWykIN8W54gYKLthKP+hOV0WpIZlNlUV2wDtSfzfB1gGD6cN/6xxendeb13Pj1NGB+gQHSMfXaIWukVt1EEUjdA7+kCfTsU5cS6c5mzUKc01+2ihnKtvfpPI1w==</latexit> q t q t ht ht 1 xt 1 <latexit sha1_base64="o6yrtT8KWAx2ZOmu7tl4jWX1C8g=">AAACh3icdVFNTwIxEC3rF+DXokcvjcTEE+6qUY6oF4+YiJrAuul2CzS227WdJZIN/8Sr/if/jV3gIKKTNHl5M6/z8iZKBTfgeV8lZ2V1bX2jXKlubm3v7Lq1vQejMk1Zhyqh9FNEDBM8YR3gINhTqhmRkWCP0ctN0X8cMW24Su5hnLJAkkHC+5wSsFTour1I5j1hBTGZhPD8Grp1r+FNCy8Dfw7qaF7tsFYKe7GimWQJUEGM6fpeCkFONHAq2KTaywxLCX0hA9a1MCGSmSCfWp/gI8vEuK+0fQngKftTkRNpzFhGdlISGJrfvYL8q9fNoN8Mcp6kGbCEzhb1M4FB4SIHHHPNKIixBYRqbr1iOiSaULBpLfx07wd5Ya74ZmF9JCfVI/yTKWykIN8W54gYKLthKP+hOV0WpIZlNlUV2wDtSfzfB1gGD6cN/6xxendeb13Pj1NGB+gQHSMfXaIWukVt1EEUjdA7+kCfTsU5cS6c5mzUKc01+2ihnKtvfpPI1w==</latexit> xt xt 1 xt <latexit sha1_base64="o6yrtT8KWAx2ZOmu7tl4jWX1C8g=">AAACh3icdVFNTwIxEC3rF+DXokcvjcTEE+6qUY6oF4+YiJrAuul2CzS227WdJZIN/8Sr/if/jV3gIKKTNHl5M6/z8iZKBTfgeV8lZ2V1bX2jXKlubm3v7Lq1vQejMk1Zhyqh9FNEDBM8YR3gINhTqhmRkWCP0ctN0X8cMW24Su5hnLJAkkHC+5wSsFTour1I5j1hBTGZhPD8Grp1r+FNCy8Dfw7qaF7tsFYKe7GimWQJUEGM6fpeCkFONHAq2KTaywxLCX0hA9a1MCGSmSCfWp/gI8vEuK+0fQngKftTkRNpzFhGdlISGJrfvYL8q9fNoN8Mcp6kGbCEzhb1M4FB4SIHHHPNKIixBYRqbr1iOiSaULBpLfx07wd5Ya74ZmF9JCfVI/yTKWykIN8W54gYKLthKP+hOV0WpIZlNlUV2wDtSfzfB1gGD6cN/6xxendeb13Pj1NGB+gQHSMfXaIWukVt1EEUjdA7+kCfTsU5cS6c5mzUKc01+2ihnKtvfpPI1w==</latexit> q t ht 1 ht xt 1 xt <latexit sha1_base64="o6yrtT8KWAx2ZOmu7tl4jWX1C8g=">AAACh3icdVFNTwIxEC3rF+DXokcvjcTEE+6qUY6oF4+YiJrAuul2CzS227WdJZIN/8Sr/if/jV3gIKKTNHl5M6/z8iZKBTfgeV8lZ2V1bX2jXKlubm3v7Lq1vQejMk1Zhyqh9FNEDBM8YR3gINhTqhmRkWCP0ctN0X8cMW24Su5hnLJAkkHC+5wSsFTour1I5j1hBTGZhPD8Grp1r+FNCy8Dfw7qaF7tsFYKe7GimWQJUEGM6fpeCkFONHAq2KTaywxLCX0hA9a1MCGSmSCfWp/gI8vEuK+0fQngKftTkRNpzFhGdlISGJrfvYL8q9fNoN8Mcp6kGbCEzhb1M4FB4SIHHHPNKIixBYRqbr1iOiSaULBpLfx07wd5Ya74ZmF9JCfVI/yTKWykIN8W54gYKLthKP+hOV0WpIZlNlUV2wDtSfzfB1gGD6cN/6xxendeb13Pj1NGB+gQHSMfXaIWukVt1EEUjdA7+kCfTsU5cS6c5mzUKc01+2ihnKtvfpPI1w==</latexit> q t r qt Let <latexit sha1_base64="F0bgirQj4yp8wQ0VHx1xRECVGIk=">AAACqnicdVHLbtNAFJ24hZbwSsuSzahRJRYoslskuqxgw4JFQU1bFBvrenyTjDozNjPXQDTyh/A1bOET+BvGaRZNA1ca6ejc15lzi1pJR3H8pxdtbd+7v7P7oP/w0eMnTwd7+xeuaqzAsahUZa8KcKikwTFJUnhVWwRdKLwsrt92+cuvaJ2szDktasw0zIycSgEUqHxwnBooFOQ+LbRPVWgsoc3p85eWp99kiSRViT7VQHMByr9vQzIfDONRvAy+CZIVGLJVnOV7vTwtK9FoNCQUODdJ4poyD5akUNj208ZhDeIaZjgJ0IBGl/nl71p+GJiSTysbniG+ZG93eNDOLXQRKjuZ7m6uI/+VmzQ0Pcm8NHVDaMTNommjOFW8s4qX0qIgtQgAhJVBKxdzsCAoGLo26TzJfCeuG7O2vtBt/5DfZjoZNenv63WgZlXYMNf/oaXYbKgdNsHVqgwGhpMkdw+wCS6ORsnx6OjDq+Hpm9VxdtlzdsBesIS9ZqfsHTtjYybYD/aT/WK/o5fRx+hTNLkpjXqrnmdsLaLyL9Tp2GI=</latexit> (a) VRNN (b) SRNN (c) SVG (d) AVF Figure 4.2: Filtering Inference Models for VRNN, SRNN, SVG, and AVF. Each diagram shows the computational graph for inferring the approximate posterior parameters, λ𝑞 , at step 𝑡. Previously proposed methods rely on hand-crafted architectures of observations, hidden states, and latent variables. AVF is a simple, general filtering procedure that only requires the local inference gradient. thors of VRNN1, SRNN2, and SVG3. Accompanying code can be found online at github.com/joelouismarino/amortized-variational-filtering. Setup Iterative inference models are implemented as specified in Eq. 4.11, encoding the approximate posterior parameters and their gradients at each inference iteration at each step. As in the static setting (Chapter 3), we separately normalize each of the inputs to the inference model using layer normalization (Ba, Kiros, and Hinton, 2016) and use highway gating (Eq. 3.12) for for each output (R. K. Srivastava, Greff, and Schmidhuber, 2015). The generative models that we evaluate contain non-spatial latent variables, thus, we use fully-connected layers to parameterize the inference models. Specifically, we use two-layer fully-connected networks with 1, 024 units per layer, internal highway connections R. K. Srivastava, Greff, and Schmidhuber, 2015, and ELU non-linearities Clevert, Unterthiner, and Hochreiter, 2015. Importantly, minimal effort went into engineering the inference model architectures: across all models and datasets, we utilize the same inference model architecture for AVF. Further model and experiment details can be found in Marino, Cvitkovic, and Yue, 2018. Finally, we note that AVF is comparable in runtime to the baseline filtering methods. For example, with our implementation of SRNN on TIMIT, AVF requires 13.1𝑚𝑠 per time step, whereas the baseline method requires 15.6𝑚𝑠 per time step. AVF requires an additional decoding per step, but inference is local to the current 1 https://github.com/jych/nips2015_vrnn 2 https://github.com/marcofraccaro/srnn 3 https://github.com/edenton/svg 66 Observation Model Output Iteration 0 Model Output Iteration 1 Figure 4.3: Prediction-Update Visualization. Test data (top), output predictions (middle), and updated reconstructions (bottom) for TIMIT using SRNN with AVF. Sequences run from left to right. The predictions made by the model already contain the general structure of the data. AVF explicitly updates the approximate posterior from the prior prediction, focusing on inference corrections rather than re-estimation. time step, making backpropagation more efficient. Speech Modeling Models For speech modeling, we use VRNN and SRNN, attempting to keep the model architectures consistent with the original implementations. The most notable difference in our implementation occurs in SRNN, where we use an LSTM rather than a GRU as the recurrent module. As in Fraccaro, S. K. Sønderby, et al., 2016, we anneal the KL divergence initially during training. In both models, we use a Gaussian output density. Unlike Chung, Kastner, et al., 2015; Fraccaro, S. K. Sønderby, et al., 2016; Goyal et al., 2017, which evaluate log densities, we evaluate and report log probabilities by integrating the output density over the data discretization window, as in modeling image pixels. Data We train and evaluate on TIMIT (Garofolo et al., 1993), which consists of audio recordings of 6, 300 sentences spoken by 630 individuals. As performed in Chung, Kastner, et al., 2015, we sample the audio waveforms at 16 kHz, split the training and validation sets into half second clips, and group each sequence into bins of 200 consecutive samples. Thus, each training and validation sequence consists of 40 model steps. Evaluation is performed on the full duration of each test sequence, averaging roughly 3 seconds. 67 Music Modeling Model We model polyphonic music using SRNN. The generative model architecture is the same as in the speech modeling experiments, with changes in the number of layers and units to match Fraccaro, S. K. Sønderby, et al., 2016. To model the binary music notes, we use a Bernoulli output distribution. Again, we anneal the KL divergence initially during training. Data We use four datasets of polyphonic (MIDI) music (Boulanger-Lewandowski, Bengio, and Vincent, 2012): Piano-midi.de, MuseData, JSB Chorales, and Nottingham. Each dataset contains between 100 and 1, 000 songs, with each song between 100 to 4, 000 steps. For training and validation, we break the sequences into clips of length 25, and we test on the entire test sequences. Video Modeling Model Our implementation of SVG differs from the original model in that we evaluate the conditional log-likelihood under a Gaussian output density rather than mean squared output error. All other architecture details are identical to the original model. However, Denton and Fergus, 2018 down-weight the KL-divergence by a factor of 10−6 at all steps. We instead remove this factor to use the ELBO during training and evaluation. As to be expected, this results in the model using the latent variables to a lesser extent. We train and evaluate SVG using filtering inference at all steps, rather than predicting multiple steps into the future, as in Denton and Fergus, 2018. Data We train and evaluate SVG on KTH Actions (Schuldt, Laptev, and Caputo, 2004), which contains 760 train / 768 val / 863 test videos of people performing various actions, each of which is between roughly 50 to 150 frames. Frames are re-sized to 64 × 64. For training and validation, we split the data into clips of 20 frames. Results Additional Inference Iterations The variational filtering EM algorithm involves inference optimization at each step (Algorithm 2, Line 6). AVF optimizes each approximate posterior through a model that learns to perform iterative updates (Eq. 4.11). Additional inference iterations ELBO Improvement (%) -ELBO per Step (nats) 68 1,160 1,150 1,140 1,130 1 2 4 Training Inference Iterations (a) 8 25 20 15 10 5 0 0 1 2 3 4 5 6 7 8 Inference Iteration (b) Figure 4.4: Improvement with Inference Iterations. Results are shown on the TIMIT validation set using VRNN with AVF. (a) Average negative ELBO per step with varying numbers of inference iterations during training. Additional iterations result in improved performance. (b) Average relative ELBO improvement from the initial (prior) estimate at each inference iteration for a single model. Empirically, each successive iteration provides diminishing additional improvement. may lead to further improvement in performance (Marino, Yue, and Mandt, 2018). We explore this aspect on TIMIT using VRNN. In Figure 4.4a, we plot the average negative ELBO per step on validation sequences for models trained with varying numbers of inference iterations. Figure 4.4b shows average relative improvement over the prior estimate for a single model trained with 8 inference iterations. We observe that training with additional inference iterations empirically leads to improved performance (Figure 4.4a), with each iteration providing diminishing improvement during inference (Figure 4.4b). This aspect is distinct from many baseline filtering methods, which directly output the approximate posterior at each step. We can also directly visualize inference improvement through the model output. Figure 4.3 illustrates example reconstructions over inference iterations, using SRNN on TIMIT. At the initial inference iteration, the approximate posterior is initialized from the prior, resulting in an output prediction. The iterative inference model then uses the approximate posterior gradients to update the estimate, improving the output reconstruction. Quantitative Comparison Tables 4.1, 4.2, and 4.3 present quantitative comparisons of average negative filtering ELBO per step (lower is better) between AVF and baseline filtering methods for TIMIT, KTH Actions, and the polyphonic music datasets respectively. On TIMIT, 69 Table 4.1: Average negative ELBO per step (in nats) on the TIMIT speech dataset for SRNN and VRNN with the respective originally proposed filtering procedures (baselines) and with AVF. Table 4.2: Average negative ELBO per step (in nats per dimension) on the KTH Actions video dataset for SVG with the originally proposed filtering procedure (baseline) and with AVF. TIMIT VRNN baseline AVF (1 Iter.) AVF (2 Iter.) SRNN baseline AVF (1 Iter.) KTH Actions SVG baseline AVF (1 Iter.) 1,082 1,105 1,071 3.69 2.86 1,026 1,024 Table 4.3: Average negative ELBO per step (in nats) on polyphonic music datasets for SRNN with and without AVF. Results from Fraccaro, S. K. Sønderby, et al., 2016 are provided for comparison, however, our model implementation differs in several aspects (see Marino, Cvitkovic, and Yue, 2018). SRNN baseline (Fraccaro et al., 2016) baseline AVF (1 Iter.) AVF (5 Iter.) Piano-midi.de MuseData JSB Chorales Nottingham 8.20 8.19 8.12 – 6.28 6.27 5.99 – 4.74 6.92 6.97 6.77 2.94 3.19 3.13 – training with AVF performs comparably to the baseline methods for both VRNN and SRNN. We note that VRNN with AVF using 2 inference iterations resulted in a final test performance of 1, 071 nats per step, outperforming the baseline method. Similar results are also observed on each of the polyphonic music datasets. Again, increasing the number of inference iterations to 5 for AVF on JSB Chorales resulted in a final test performance of 6.77 nats per step. AVF significantly improves the performance of SVG on KTH Actions. We attribute this, likely, to the absence of the KL down-weighting factor in our training objective as compared with (Denton and Fergus, 2018). This agrees with Kumar et al., 2020, who observe that many recently proposed video models trained with proper log-likelihood objectives tend to perform poorly. The baseline filtering procedure seems to struggle to a greater degree than AVF. From comparing the results above, we see that AVF is a general filtering procedure that performs well across multiple models and datasets, despite using a relatively simple inference model structure. In the cases where single- 70 step AVF did not outperform the baseline inference procedure, we were able to improve performance by simply increasing the number of inference iterations, with no additional parameter overhead. 4.5 Discussion This chapter introduced the variational filtering EM algorithm for performing filtering inference in sequential latent variable models. Filtering variational inference can be expressed as a sequence of optimization objectives, linked across steps through previous latent samples. Using iterative amortized inference (Chapter 3) to perform inference optimization, we arrived at an efficient implementation of the algorithm: amortized variational filtering. This general filtering method scales to large models and datasets. Numerous methods have been proposed for filtering in deep sequential latent variable models, with each method hand–designed for each model. The variational filtering EM algorithm provides a single framework for analyzing and constructing these methods. Amortized variational filtering is a simple, theoretically-motivated, and general filtering method that we have shown performs on-par with or better than multiple existing state-of-the-art methods across various data domains. Unlike previous amortized filtering methods, AVF is formulated in terms of iterative optimization at each time step. That is, AVF is initialized at each time step from the prior and only optimizes the relevant objective terms at the current time step. As a result, AVF does not need to re-estimate the latent dynamics, as in previous amortized filtering methods. This prediction-update scheme is reminiscent of classical Bayesian filtering techniques, such as Kalman filtering (Kalman, 1960), using prediction errors to perform exact probabilistic inference in linear-Gaussian models. From this perspective, AVF is thus a modern instantiation of the same negative feedback principles for perception, harnessing the efficiency of amortization, combined with the generality of variational inference. 4.6 Appendix: Filtering ELBO Derivation This derivation largely follows that of Gemici et al., 2017 and is valid for any filtering approximate posterior. From Eq. 4.3, we have the definition of the ELBO:   𝑝 𝜃 (x≤𝑇 , z≤𝑇 ) L ≡ E𝑞(z ≤𝑇 |x ≤𝑇 ) log . (4.12) 𝑞(z≤𝑇 |x≤𝑇 ) 71 Plugging in the forms of the joint distribution (Eq. 4.1) and approximate posterior (Eq. 4.4), we can write the term within the expectation as a sum: " !# 𝑇 Ö 𝑝(x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) L =E𝑞(z ≤𝑇 |x ≤𝑇 ) log (4.13) 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) 𝑡=1 " 𝑇 # Õ 𝑝(x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) log L =E𝑞(z ≤𝑇 |x ≤𝑇 ) (4.14) 𝑞(z |x , z ) 𝑡 ≤𝑡 <𝑡 𝑡=1 " 𝑇 # Õ L =E𝑞(z ≤𝑇 |x ≤𝑇 ) 𝐶𝑡 , (4.15) 𝑡=1 where the term 𝐶𝑡 is defined to simplify notation. We then expand the expectation: " 𝑇 # Õ L = E𝑞(z1 |x1 ) . . . E𝑞(z𝑇 |x ≤𝑇 ,z<𝑇 ) 𝐶𝑡 (4.16) 𝑡=1 There are 𝑇 terms within the sum, but each 𝐶𝑡 only depends on the expectations up to time 𝑡 because we only condition on past and present variables. This allows us to write: L =E𝑞(z1 |x1 ) [𝐶1 ] + E𝑞(z1 |x1 ) E𝑞(z2 |x ≤2 ,z1 ) [𝐶2 ] +... + E𝑞(z1 |x1 ) E𝑞(z2 |x ≤2 ,z1 ) . . . E𝑞(z𝑇 |,x ≤𝑇 ,z<𝑇 ) [𝐶𝑇 ] L= L= L= 𝑇 Õ 𝑡=1 𝑇 Õ 𝑡=1 𝑇 Õ (4.17) E𝑞(z ≤𝑡 |x ≤𝑡 ) [𝐶𝑡 ] (4.18) EÎ𝑡𝜏=1 𝑞(z 𝜏 |x ≤𝜏 ,z<𝜏 ) [𝐶𝑡 ] (4.19)   EÎ𝑡−1 𝑞(z 𝜏 |x ≤𝜏 ,z<𝜏 ) E𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) [𝐶𝑡 ] . (4.20) 𝜏=1 𝑡=1 As in Section 4.2, we define L𝑡 as L𝑡 ≡ E𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) [𝐶𝑡 ]   𝑝 𝜃 (x𝑡 , z𝑡 |x<𝑡 , z<𝑡 ) L𝑡 = E𝑞(z𝑡 |x ≤𝑡 ,z<𝑡 ) log . 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) (4.21) (4.22) This allows us to write Eq. 4.20 as L= 𝑇 Õ EÎ𝑡−1 𝑞(z 𝜏 |x ≤𝜏 ,z<𝜏 ) [L𝑡 ] , 𝜏=1 𝑡=1 which agrees with Eq. 4.5. (4.23) 72 References Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E Hinton (2016). “Layer normalization”. In: arXiv preprint arXiv:1607.06450. Babaeizadeh, Mohammad et al. (2018). “Stochastic Variational Video Prediction”. In: International Conference on Learning Representations. Bayer, Justin and Christian Osendorfer (2014). “Learning Stochastic Recurrent Networks”. In: NeurIPS 2014 Workshop on Advances in Variational Inference. Boulanger-Lewandowski, Nicolas, Yoshua Bengio, and Pascal Vincent (2012). “Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription”. In: International Conference on Machine Learning. Chung, Junyoung, Caglar Gulcehre, et al. (2014). “Empirical evaluation of gated recurrent neural networks on sequence modeling”. In: arXiv preprint arXiv:1412.3555. Chung, Junyoung, Kyle Kastner, et al. (2015). “A recurrent latent variable model for sequential data”. In: Advances in neural information processing systems, pp. 2980– 2988. Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter (2015). “Fast and accurate deep network learning by exponential linear units (elus)”. In: arXiv preprint arXiv:1511.07289. Denton, Emily and Rob Fergus (2018). “Stochastic Video Generation with a Learned Prior”. In: International Conference on Machine Learning, pp. 1182–1191. Finn, Chelsea, Ian Goodfellow, and Sergey Levine (2016). “Unsupervised learning for physical interaction through video prediction”. In: Advances in Neural Information Processing Systems. Fraccaro, Marco, Simon Kamronn, et al. (2017). “A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning”. In: Advances in Neural Information Processing Systems. Fraccaro, Marco, Søren Kaae Sønderby, et al. (2016). “Sequential neural models with stochastic layers”. In: Advances in neural information processing systems, pp. 2199–2207. Friston, Karl (2005). “A theory of cortical responses”. In: Philosophical Transactions of the Royal Society of London B: Biological Sciences 360.1456, pp. 815–836. Garofolo, J. S. et al. (1993). DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus. Gemici, Mevlana et al. (2017). “Generative Temporal Models with Memory”. In: arXiv preprint arXiv:1702.04649. Gershman, Samuel and Noah Goodman (2014). “Amortized inference in probabilistic reasoning”. In: Proceedings of the Cognitive Science Society. Vol. 36. 36. 73 Goyal, Anirudh et al. (2017). “Z-Forcing: Training Stochastic Recurrent Networks”. In: Advances in Neural Information Processing Systems. He, Jiawei, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal (2018). “Probabilistic video generation using holistic attribute control”. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 452–467. url: http://openaccess.thecvf.com/content_ECCV_2018/html/Jiawei_ He_Probabilistic_Video_Generation_ECCV_2018_paper.html. Henaff, Mikael, Junbo Zhao, and Yann LeCun (2017). “Prediction Under Uncertainty with Error-Encoding Networks”. In: arXiv preprint arXiv:1711.04994. Hoffman, Matthew D et al. (2013). “Stochastic variational inference”. In: The Journal of Machine Learning Research 14.1, pp. 1303–1347. Hsu, Wei-Ning, Yu Zhang, and James Glass (2017). “Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data”. In: Advances in Neural Information Processing Systems. Johnson, Matthew et al. (2016). “Composing graphical models with neural networks for structured representations and fast inference”. In: Advances in Neural Information Processing Systems. Jordan, Michael I et al. (1998). “An introduction to variational methods for graphical models”. In: NATO ASI SERIES D BEHAVIOURAL AND SOCIAL SCIENCES 89, pp. 105–162. Kalman, Rudolph Emil (1960). “A new approach to linear filtering and prediction problems”. In: Journal of Basic Engineering 82.1, pp. 35–45. Karl, Maximilian et al. (2017). “Deep variational Bayes filters: Unsupervised learning of state space models from raw data”. In: Proceedings of the International Conference on Learning Representations (ICLR). Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Kumar, Manoj et al. (2020). “VideoFlow: A Flow-Based Generative Model for Video”. In: International Conference on Learning Representations. Le, Tuan Anh et al. (2018). “Auto-Encoding Sequential Monte Carlo”. In: International Conference on Learning Representations. Li, Yingzhen and Stephan Mandt (2018). “A Deep Generative Model for Disentangled Representations of Sequential Data”. In: International Conference on Machine Learning. Lotter, William, Gabriel Kreiman, and David Cox (2017). “Deep predictive coding networks for video prediction and unsupervised learning”. In: International Conference on Learning Representations. 74 Maddison, Chris J et al. (2017). “Filtering Variational Objectives”. In: Advances in Neural Information Processing Systems. Marino, Joseph, Milan Cvitkovic, and Yisong Yue (2018). “A general method for amortizing variational filtering”. In: Advances in Neural Information Processing Systems, pp. 7857–7868. url: http://papers.nips.cc/paper/8011- ageneral-method-for-amortizing-variational-filtering. Marino, Joseph, Yisong Yue, and Stephan Mandt (2018). “Iterative Amortized Inference”. In: International Conference on Machine Learning, pp. 3403–3412. url: http://proceedings.mlr.press/v80/marino18a.html. Murphy, Kevin P (2012). Machine learning: a probabilistic perspective. MIT press. Naesseth, Christian et al. (2018). “Variational Sequential Monte Carlo”. In: International Conference on Artificial Intelligence and Statistics. Neal, Radford M and Geoffrey E Hinton (1998). “A view of the EM algorithm that justifies incremental, sparse, and other variants”. In: Learning in graphical models. Springer, pp. 355–368. Ranganath, Rajesh, Sean Gerrish, and David Blei (2014). “Black box variational inference”. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 814–822. Rao, Rajesh PN and Dana H Ballard (1999). “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” In: Nature neuroscience 2.1. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Särkkä, Simo (2013). Bayesian filtering and smoothing. Vol. 3. Cambridge University Press. Schuldt, Christian, Ivan Laptev, and Barbara Caputo (2004). “Recognizing human actions: a local SVM approach”. In: International Conference on Pattern Recognition. Sønderby, Casper Kaae et al. (2016). “Ladder variational autoencoders”. In: Advances in Neural Information Processing Systems (NIPS), pp. 3738–3746. Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov (2015). “Unsupervised learning of video representations using lstms”. In: International conference on machine learning, pp. 843–852. Srivastava, Rupesh K, Klaus Greff, and Jürgen Schmidhuber (2015). “Training very deep networks”. In: Advances in neural information processing systems (NIPS), pp. 2377–2385. 75 Walker, Jacob et al. (2016). “An uncertain future: Forecasting from static images using variational autoencoders”. In: European Conference on Computer Vision. Xue, Tianfan et al. (2016). “Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks”. In: Advances in Neural Information Processing Systems (NIPS), pp. 91–99. 76 Chapter 5 IMPROVING SEQUENTIAL LATENT VARIABLE MODELS WITH AUTOREGRESSIVE FLOWS learned feedforward perception 13 Marino, Joseph, Lei Chen, Jiawei He, and Stephan Mandt (2020). “Improving Sequential Latent Variable Models with Autoregressive Flows”. In: Symposium on Advances in Approximate Bayesian Inference, pp. 1–16. url: http : / / proceedings.mlr.press/v118/marino20a.html. Yang, Ruihan, Yibo Yang, Joseph Marino, and Stephan Mandt (2021). “Hierarchical Autoregressive Modeling for Neural Video Compression”. In: International Conference on Learning Representations. url: https://openreview.net/ pdf?id=TK_6nNb_C7q. 5.1 Introduction The previous chapters explored the use of learned negative feedback to perform inference in deep latent variable models. These methods attempt to reduce errors, via variational inference optimization, as they arise. As we saw in Chapter 4, inference optimization can be augmented with prior predictions, providing a useful initialization that preemptively reduces errors. However, these priors occur within the latent space, which must be transformed into data-level predictions at each time step. That is, the entire raw data input is constantly being newly predicted. This chapter explores a complementary technique, incorporating predictions directly at the data level, via the conditional likelihood, thereby completely bypassing inference 77 optimization. In this way, low-level predictions remove temporal redundancy from the data, with latent variables, and therefore inference optimization, only reserved for any remaining errors. We formulate this process as an autoregressive flow across time, yielding a form of learned feedforward perception. Feedforward processing can be seen as learning a frame of reference to assist in modeling the data by pre-processing the input at each time step. This not only simplifies downstream dynamics modeling, but, by modeling errors rather than the data itself, also has the effect of improving generalization to unseen sequences. We demonstrate these improvements across multiple benchmark video datasets. 5.2 Method We start by motivating the use of autoregressive flows to reduce temporal dependencies, thereby simplifying dynamics. We then show how this simple technique can be incorporated within sequential latent variable models. Motivation: Temporal Redundancy Reduction Normalizing flows, while often utilized for density estimation, originated from data pre-processing techniques (Friedman, 1987; Hyvärinen and Oja, 2000; S. S. Chen and Gopinath, 2001), which remove dependencies between dimensions, i.e., redundancy reduction (Barlow et al., 1961). Removing dependencies simplifies the resulting probability distribution by restricting variation to individual dimensions, generally simplifying downstream tasks (Laparra, Camps-Valls, and Malo, 2011). Normalizing flows improve upon these procedures using flexible, non-linear functions (Deco and Brauer, 1995; Dinh, Krueger, and Bengio, 2015). While flows have been used for spatial decorrelation (Agrawal and Dukkipati, 2016; Winkler et al., 2019) and with other models (Huang et al., 2017), this capability remains under-explored. Data sequences contain dependencies in time, for example, in the redundancy of video pixels (Figure 5.3), which are often highly predictable. These dependencies define the dynamics of the data, with the degree of dependence quantified by the multi-information, Õ I (x1:𝑇 ) = H (x𝑡 ) − H (x1:𝑇 ), (5.1) 𝑡 where H denotes entropy. Normalizing flows are capable of reducing redundancy, arriving at a new sequence, y1:𝑇 , with I (y1:𝑇 ) ≤ I (x1:𝑇 ), thereby reducing temporal dependencies and simplifying dynamics. Thus, rather than fit the data distribution directly, we can first simplify the dynamics by pre-processing sequences with a 0.6 0.6 0.6 0.6 p(x2|x1) 0.4 p(y2|y1) 78 0.4 0.2 0.0 −5.0 −2.5 0.0 x2 (a) 2.5 5.0 0.4 0.2 0.0 −5.0 p(y2) p(x2) 0.4 0.2 −2.5 0.0 x2 2.5 5.0 0.0 −5.0 −2.5 (b) 0.0 y2 2.5 5.0 0.2 0.0 −5.0 (c) −2.5 0.0 y2 2.5 5.0 (d) Figure 5.1: Redundancy Reduction. (a) Conditional densities for 𝑝(𝑥 2 |𝑥 1 ). (b) The marginal, 𝑝(𝑥 2 ) differs from the conditional densities, thus, I (𝑥 1 ; 𝑥 2 ) > 0. (c) In the normalized space of 𝑦, the corresponding densities 𝑝(𝑦 2 |𝑦 1 ) are identical. (d) The marginal 𝑝(𝑦 2 ) is identical to the conditionals, so I (𝑦 1 ; 𝑦 2 ) = 0. Thus, in this case, a conditional affine transform removed the dependencies. normalizing flow, then fit the resulting sequence. Through training, the flow will attempt to remove redundancies to meet the modeling capacity of the higher-level dynamics model, 𝑝 𝜃 (y1:𝑇 ). Example To visualize this procedure for an affine autoregressive flow, consider a one-dimensional input over two time steps, 𝑥 1 and 𝑥 2 . For each value of 𝑥1 , there is a conditional density, 𝑝(𝑥 2 |𝑥 1 ). Assume that these densities take one of two forms, which are identical but shifted and scaled, shown in Figure 5.1. Transforming these densities through their conditional means, 𝜇2 = E [𝑥 2 |𝑥 1 ], and standard deviations,   1/2 𝜎2 = E (𝑥 2 − 𝜇2 ) 2 |𝑥 1 , creates a normalized space, 𝑦 2 = (𝑥 2 − 𝜇2 )/𝜎2 , where the conditional densities are identical. In this space, the multi-information is I (𝑦 1 ; 𝑦 2 ) = E 𝑝(𝑦1 ,𝑦2 ) [log 𝑝(𝑦 2 |𝑦 1 ) − log 𝑝(𝑦 2 )] = 0, whereas I (𝑥 1 ; 𝑥 2 ) > 0. Indeed, if 𝑝(𝑥𝑡 |𝑥 <𝑡 ) is linear-Gaussian, inverting an affine autoregressive flow exactly corresponds to Cholesky whitening (Pourahmadi, 2011; Kingma, Salimans, et al., 2016), removing all linear dependencies. In the example above, 𝜇2 and 𝜎2 act as a frame of reference for estimating 𝑥 2 . More generally, in the special case where µ𝜃 (x<𝑡 ) = x𝑡−1 and σ(x<𝑡 ) = 1, we recover y𝑡 = x𝑡 −x𝑡−1 = Δx𝑡 . Modeling finite differences (or generalized coordinates (Friston, 2008)) is a well-established technique, (see, e.g., (Chua et al., 2018; Kumar et al., 2020)), which is generalized by affine autoregressive flows. Modeling Dynamics with Autoregressive Flows We now discuss utilizing autoregressive flows to improve sequence modeling, highlighting use cases for modeling dynamics in the data and latent spaces. 79 Sequential Latent Variable <latexit sha1_base64="//Wehlur2MY6EVVm2lvbA5j++Zw=">AAACinicdVHLSgMxFE3Hd31VXboJFsFVmamCipuiLlwqtlVoh5JJb9tgMhmTO2IZ+i1u9ZP8GzO1iz70woXDua/DuVEihUXf/y54S8srq2vrG8XNre2d3dLeftPq1HBocC21eY6YBSliaKBACc+JAaYiCU/Ry01ef3oDY4WO6zhMIFSsH4ue4Awd1SkdtBHe0ajsEV5TiFEwOeqUyn7FHwddBMEElMkk7jt7hU67q3mq3AIumbWtwE8wzJhBwSWMiu3UQsL4C+tDy8GYKbBhNlY/oseO6dKeNi5jpGN2eiJjytqhilynYjiw87Wc/KvWSrF3EWYiTlKEmP8e6qWSoqa5FbQrDHCUQwcYN8JppXzADOPoDJvZVA/CLBeXr5k5H6lR8ZhOM7mMBNX7bB+Tfe0uDNQ/tOCLA4mF1Lmqu85A95Jg/gGLoFmtBKeV6sNZuXY9ec46OSRH5IQE5JzUyB25Jw3CyZB8kE/y5W15Ve/Su/pt9QqTmQMyE97tD0YWyoY=</latexit> Model <latexit sha1_base64="GapF9uF0mvCTWqE3sPAHUh/6PLE=">AAACj3icdVHBSiNBEO2Mu65mXU3Wi+Cl2SB4CjPugp4kuBcFDxFMFJIh1HQ6SWP39NBdsxiG+DVe9X/2b7YmycEYt6Dh8apeVfWrJNPKYxj+rQQbnz5vftnarn7d+ba7V6t/73qbOyE7wmrr7hPwUqtUdlChlveZk2ASLe+Sh99l/u6PdF7Z9BanmYwNjFM1UgKQqEHtoI/yEZ0prgFlirwLTgGJZ4NaI2yG8+DrIFqCBltGe1CvDPpDK3JDbYQG73tRmGFcgEMlqGG1n3uZgXiAsewRTMFIHxfzL8z4ETFDPrKOHq0xZ98qCjDeT01ClQZw4t/nSvKjXC/H0VlcqDTL6X9iMWiUa46Wl37woXJSoJ4SAOEU7crFBBwIJNdWOt1GcVEuV7ZZGZ+YWfWIv2XKNTI0j6t1oMeWJkzMf2gl1gWZlzm5aodkIJ0ken+AddA9aUY/myc3vxqti+Vxttgh+8GOWcROWYtdsjbrMMGe2DN7Ya9BPTgNzoPWojSoLDX7bCWCq39tJMxL</latexit> <latexit sha1_base64="CzgMrFKkGOsa2Onmbgn1zXOc4xo=">AAACg3icdVHLSgMxFE3HV61vXboJFkEQykwt6EYQ3bgRKthWaMeSSW/bYDIZkjtiGeY/3Opf+TdmahfW6oULh3Nfh3uiRAqLvv9Z8paWV1bXyuuVjc2t7Z3dvf221anh0OJaavMYMQtSxNBCgRIeEwNMRRI60fNNUe+8gLFCxw84SSBUbBSLoeAMHfXUQ3hFo7I7PQCZ93erfs2fBl0EwQxUySya/b1SvzfQPFUQI5fM2m7gJxhmzKDgEvJKL7WQMP7MRtB1MGYKbJhNZef02DEDOtTGZYx0yv6cyJiydqIi16kYju3vWkH+VeumOLwIMxEnKULMvw8NU0lR0+IHdCAMcJQTBxg3wmmlfMwM4+g+NbfpIQizQlyxZu58pPLKMf3JFDISVK/zfUyOtLswVv/Qgi8OJBZS91VnSV5xlgS/DVgE7XotOKvV7xvVq+uZOWVySI7ICQnIObkit6RJWoQTQ97IO/nwVrxTr+41vlu90mzmgMyFd/kFY4rH+Q==</latexit> Autoregressive Flow <latexit sha1_base64="ZKOsX4nA+FCRGgg3ahsu1NSlNPA=">AAACjnicdVHLSgMxFE3Hd321unDhJlgEV2VGRd2IVTcuK7QqtEPJpLdtMJkMyR2xDPM1bvWD/BsztQtr9ULgcO7j3JwbJVJY9P3PkrewuLS8srpWXt/Y3NquVHcerE4NhzbXUpuniFmQIoY2CpTwlBhgKpLwGD3fFvnHFzBW6LiF4wRCxYaxGAjO0FG9yl4X4RWNyq5T1AaGBqwVL5D3KjW/7k+CzoNgCmpkGs1etdTr9jVPFcTIJbO2E/gJhhkzKLiEvNxNLSSMP7MhdByMmQIbZpMf5PTQMX060Ma9GOmE/dmRMWXtWEWuUjEc2d+5gvwr10lxcBFmIk5ShJh/Cw1SSVHTwg7aFwY4yrEDjBvhdqV8xAzj6EybmdQKwqxYrhgzIx+pvHxIfzLFGgmq19k6JofaKYzUP7Tg8w2JhdS5qvvOQHeS4PcB5sHDcT04qR/fn9YaN9PjrJJ9ckCOSEDOSYPckSZpE05y8kbeyYdX8c68S+/qu9QrTXt2yUx4d1+ZJcxr</latexit> <latexit sha1_base64="9c2S84X+Yjx7lgHsBqZAnAQM7c8=">AAACgnicdVHLSgMxFE3Hd321unQTLIK4KDNW0IWLoiAuK1gV2qFk0ts2mEzG5I62DP0Ot/pZ/o2Ztgtr9cKFw7mvw7lRIoVF3/8qeEvLK6tr6xvFza3tnd1See/B6tRwaHIttXmKmAUpYmiiQAlPiQGmIgmP0fN1Xn98BWOFju9xlECoWD8WPcEZOipsIwzRqOxG6rdxp1Txq/4k6CIIZqBCZtHolAuddlfzVEGMXDJrW4GfYJgxg4JLGBfbqYWE8WfWh5aDMVNgw2yiekyPHNOlPW1cxkgn7M+JjClrRypynYrhwP6u5eRftVaKvYswE3GSIsR8eqiXSoqa5hbQrjDAUY4cYNwIp5XyATOMozNqbtN9EGa5uHzN3PlIjYtH9CeTy0hQDef7mOxrd2Gg/qEFXxxILKTOVd11BrqXBL8fsAgeTqtBrXp6d1apX82es04OyCE5JgE5J3VySxqkSTh5Ie/kg3x6y96JF3i1aatXmM3sk7nwLr8BgEbHlg==</latexit> (Inference) <latexit sha1_base64="xVZXZbpac/CCiRQmU/qW9oZB2m4=">AAACi3icdVHLihNBFK20OsbMK6Pu3BSGwLgJ3ZkBRVwMI4LuRpg8IGma6srtpEg9mqrbMrHJv7gd/8i/sTrpxWSiFwoO575OnZvmUjgMwz+N4MnTZwfPmy9ah0fHJ6fts5dDZwrLYcCNNHacMgdSaBigQAnj3AJTqYRRuvxc5Uc/wDph9C2ucogVm2uRCc7QU0n79RThDq0qz7/pDCxoDu/WSbsT9sJN0H0Q1aBD6rhJzhrJdGZ4oUAjl8y5SRTmGJfMouAS1q1p4SBnfMnmMPFQMwUuLjfy17TrmRnNjPVPI92wDztKppxbqdRXKoYL9zhXkf/KTQrMPsSl0HmB/mPbRVkhKRpaeUFnwgJHufKAcSu8VsoXzDKO3rGdSbdRXFbiqjE761O1bnXpQ6aSkaO6261jcm78hoX6Dy34fkPuoPCumpk30J8kenyAfTDs96KLXv/7Zefquj5Ok7whb8k5ich7ckW+khsyIJz8JL/IPfkdHAcXwcfg07Y0aNQ9r8hOBF/+AvWqylU=</latexit> ⇢ ⇢ zt 2 zt 1 zt zt+1 zt+2 yt <latexit sha1_base64="Rc8skmreVThEy0HMt+P8j5ZBRTE=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxqWBV6AxDJs20ockkJHfEMvQ33Opv+TdmahfW6oULh3Nfh3MTLbgF3/+seNW19Y3N2lZ9e2d3b7/RPHi0KjeU9akSyjwnxDLBM9YHDoI9a8OITAR7SiY3Zf3phRnLVfYAU80iSUYZTzkl4KgwlATGSVpMZzHEjZbf8eeBV0GwAC20iLu4WYnDoaK5ZBlQQawdBL6GqCAGOBVsVg9zyzShEzJiAwczIpmNirnoGW47ZohTZVxmgOfsz4mCSGunMnGdpUj7u1aSf9UGOaSXUcEznQPL6PehNBcYFC4dwENuGAUxdYBQw51WTMfEEArOp6VND0FUlOLKNUvnEzmrt/FPppShQb4u9xExUu7CWP5Dc7o6oC3Lnatq6Ax0Lwl+P2AVPHY7wVmne3/e6l0vnlNDR+gYnaAAXaAeukV3qI8o0ugNvaMPr+qder7X/W71KouZQ7QU3tUXUArHDg==</latexit> <latexit sha1_base64="tuXXMIg/Bc4YouWi1xyioUAsaLE=">AAACm3icdVHLTttAFJ24tIX0QaDLqtKIFIluIhuQ2gULBJuqYkElAkixZV1PrpMRMx5r5roisvwD/Rq27Z/0bzoOWRACVxrp6NzXmXOzUklHYfivE7xYe/nq9fpG983bd+83e1vbl85UVuBQGGXsdQYOlSxwSJIUXpcWQWcKr7Kb0zZ/9Qutk6a4oFmJiYZJIXMpgDyV9j7Hma5jJycamjSmKRLwvVgDTbO8vm3S+oiaL2mvHw7CefBVEC1Any3iPN3qpPHYiEpjQUKBc6MoLCmpwZIUCptuXDksQdzABEceFqDRJfX8Ow3f9cyY58b6VxCfsw87atDOzXTmK1uh7nGuJZ/KjSrKvyW1LMqKsBD3i/JKcTK89YaPpUVBauYBCCu9Vi6mYEGQd3Bp0kWU1K24dszS+kw33V3+kGlllKRvl+tATYzfMNXP0FKsNpQOK++qGXsD/UmixwdYBZf7g+hgsP/zsH98sjjOOvvIdtgei9hXdsy+s3M2ZIL9ZnfsD/sbfApOgx/B2X1p0Fn0fGBLEQz/A+6F0TQ=</latexit> ✓ (x<t ) ÷ <latexit sha1_base64="zU1t23bQbWEUhSwGFZ+dhUDLseE=">AAACeXicdVHLTgIxFO2ML8QX6NJNlZAQF2QGTHRJdOMSE14JTEin04HGdjppO0Qy4Rfc6q/5LW7sAAsG9CZNTs59nNN7/ZhRpR3n27L39g8OjwrHxZPTs/OLUvmyp0QiMeliwYQc+EgRRiPS1VQzMoglQdxnpO+/PWf5/oxIRUXU0fOYeBxNIhpSjHRGjQI6G5cqTt1ZBtwF7hpUwDra47I1HgUCJ5xEGjOk1NB1Yu2lSGqKGVkUR4kiMcJvaEKGBkaIE+WlS7MLWDVMAEMhzYs0XLKbHSniSs25byo50lO1ncvIv3LDRIePXkqjONEkwiuhMGFQC5j9HAZUEqzZ3ACEJTVeIZ4iibA2+8lN6rhempnLxuTkfb4oVuEmk9mINX/P1yE2EUZhyv+hKd5tiBVJzFZFYBZoTuJuH2AX9Bp1t1lvvN5XWk/r4xTANbgFNeCCB9ACL6ANugCDKfgAn+DL+rFv7Jp9tyq1rXXPFciF3fwFDZDESg==</latexit> yt 2 yt 1 yt yt+1 yt+2 cat <latexit sha1_base64="YlEnZegu6BMRciM97uIB7y4yLos=">AAACdnicdVHLTgIxFC3jC/EFujQxjYToRjKDJrokunEJCa8EJqRTLtDQTidtx0gmfIFb/Tj/xKUdYMFDb9Lk5NzHOb03iDjTxnW/M87O7t7+QfYwd3R8cnqWL5y3tIwVhSaVXKpOQDRwFkLTMMOhEykgIuDQDiYvab79BkozGTbMNAJfkFHIhowSY6n6XT9fdMvuPPA28JagiJZR6xcy/d5A0lhAaCgnWnc9NzJ+QpRhlMMs14s1RIROyAi6FoZEgPaTudMZLllmgIdS2RcaPGdXOxIitJ6KwFYKYsZ6M5eSf+W6sRk++QkLo9hASBdCw5hjI3H6bTxgCqjhUwsIVcx6xXRMFKHGLmdtUsPzk9RcOmZNPhCzXAmvMqmNyIj39TrCR9IqjMU/NKPbDZGG2G5VDuwC7Um8zQNsg1al7N2XK/WHYvV5eZwsukTX6BZ56BFV0SuqoSaiCNAH+kRfmR/nyik5N4tSJ7PsuUBr4bi/nK/Cug==</latexit> <latexit sha1_base64="3CFBwyotL035Oaq7NDjfbjjQZ9Q=">AAACgXicdVHLSgMxFE3Hd31VXboJFkE3ZaYKCm5ENy4VbCu0Q8mkt20wmYTkjrQM/Q23+lv+jZnahW31QuDk3Nfh3MRI4TAMv0rByura+sbmVnl7Z3dvv3Jw2HQ6sxwaXEttXxLmQIoUGihQwouxwFQioZW83hf51htYJ3T6jGMDsWKDVPQFZ+ipTgdhhIi5/066lWpYC6dBl0E0A1Uyi8fuQanb6WmeKUiRS+ZcOwoNxjmzKLiESbmTOTCMv7IBtD1MmQIX51PRE3rqmR7ta+tfinTK/u7ImXJurBJfqRgO3WKuIP/KtTPsX8e5SE2GkPKfRf1MUtS0cID2hAWOcuwB41Z4rZQPmWUcvU9zk56jOC/EFWPm1idqUj6lv5lChkE1mq9jcqD9hqH6hxZ8ucE4yLyruucN9CeJFg+wDJr1WnRRqz9dVm/vZsfZJMfkhJyRiFyRW/JAHkmDcGLIO/kgn8FKcB6EQf2nNCjNeo7IXAQ336HzxzU=</latexit> µ✓ (x<t ) <latexit sha1_base64="wy1WYPqqnkkv22S/qgVxrO+jOM4=">AAACmHicdVFNT9tAEN24tNC0lAC39rJthEQvkQ1I9MAhggP0BogAUmxZ6804WbHrtXbHVSLL5/4arvBb+DeskxwIgZFWenrz9fZNkkth0fefGt6HlY+fVtc+N798Xf+20drcura6MBx6XEttbhNmQYoMeihQwm1ugKlEwk1yd1Lnb/6BsUJnVzjJIVJsmIlUcIaOils/w0SVoSqqOMQRIKO7oWI4StJyXMXlEVa/41bb7/jToMsgmIM2mcd5vNmIw4HmhYIMuWTW9gM/x6hkBgWXUDXDwkLO+B0bQt/BjCmwUTn9S0V3HDOgqTbuZUin7MuOkilrJypxlbVQ+zpXk2/l+gWmf6JSZHmBkPHZorSQFDWtjaEDYYCjnDjAuBFOK+UjZhhHZ9/CpKsgKmtx9ZiF9Ymqmjv0JVPLyFGNF+uYHGq3YaTeoQVfbsgtFM5VPXAGupMErw+wDK73OsF+Z+/ioN09nh9njfwgv8guCcgh6ZIzck56hJP/5J48kEfvu9f1Tr2/s1KvMe/ZJgvhXT4D4PDP5w==</latexit> xt 2 xt 1 xt xt+1 xt+2 xt 3 <latexit sha1_base64="A+NwkXFLy8dEXhO4lC/Y2RBrA7Y=">AAACh3icdVHLSsNAFJ3GZ+sr6tLNYBHcWJMq2qWPjUsFq0IbwmQ6aQdnMmHmRlpC/sSt/pN/46Ttwli9cOFw7utwbpQKbsDzvmrO0vLK6tp6vbGxubW94+7uPRmVacq6VAmlXyJimOAJ6wIHwV5SzYiMBHuOXm/L+vMb04ar5BEmKQskGSY85pSApULX7UsCoyjOx0WYw8lZEbpNr+VNAy8Cfw6aaB734W4t7A8UzSRLgApiTM/3UghyooFTwYpGPzMsJfSVDFnPwoRIZoJ8Kr3AR5YZ4FhpmwngKftzIifSmImMbGcp1PyuleRftV4GcSfIeZJmwBI6OxRnAoPCpQ94wDWjICYWEKq51YrpiGhCwbpV2fToB3kprlxTOR/JonGEfzKljBTkuNpHxFDZCyP5D83p4kBqWGZdVQNroH2J//sBi+Cp3fLPWu2H8+bVzfw56+gAHaJj5KNLdIXu0D3qIore0Dv6QJ9O3Tl1LpzOrNWpzWf2USWc629L3Mi+</latexit> (a) xt 2 <latexit sha1_base64="FQVlMEFQvZFJH0/AEMcNlsO9I7E=">AAACh3icdVHLTsJAFB3qC/AFunQzkZC4EVs06tLHxiUmoibQNNNhChNnOs3MrYE0/RO3+k/+jVNgIaA3ucnJua+Tc8NEcAOu+11y1tY3NrfKler2zu7efq1+8GxUqinrUiWUfg2JYYLHrAscBHtNNCMyFOwlfLsv6i/vTBuu4ieYJMyXZBjziFMClgpqtb4kMAqjbJwHGZy286DWcFvuNPAq8OaggebRCeqloD9QNJUsBiqIMT3PTcDPiAZOBcur/dSwhNA3MmQ9C2MimfGzqfQcNy0zwJHSNmPAU/b3REakMRMZ2s5CqFmuFeRftV4K0bWf8ThJgcV0dihKBQaFCx/wgGtGQUwsIFRzqxXTEdGEgnVrYdOT52eFuGLNwvlQ5tUm/s0UMhKQ48U+IobKXhjJf2hOVwcSw1LrqhpYA+1LvOUHrILndss7b7UfLxo3d/PnlNEROkYnyENX6AY9oA7qIore0Qf6RF9OxTlzLp3rWatTms8cooVwbn8AScnIvQ==</latexit> xt 1 <latexit sha1_base64="Uoe094Ebd5E0MjBKCabrmYfr41Y=">AAACh3icdVHJTsMwEHXD1patwJGLRYXEhZIUBBxZLhxB6ia1UeS4Tmthx5E9qaii/AlX+Cf+Bqf00FIYaaSnN9vTmzAR3IDrfpWctfWNza1ypbq9s7u3Xzs47BiVasraVAmleyExTPCYtYGDYL1EMyJDwbrh62NR706YNlzFLZgmzJdkFPOIUwKWCmq1gSQwDqPsLQ8yOPfyoFZ3G+4s8Crw5qCO5vEcHJSCwVDRVLIYqCDG9D03AT8jGjgVLK8OUsMSQl/JiPUtjIlkxs9m0nN8apkhjpS2GQOesYsTGZHGTGVoOwuh5netIP+q9VOIbv2Mx0kKLKY/h6JUYFC48AEPuWYUxNQCQjW3WjEdE00oWLeWNrU8PyvEFWuWzocyr57iRaaQkYB8W+4jYqTshbH8h+Z0dSAxLLWuqqE10L7E+/2AVdBpNrzLRvPlqn73MH9OGR2jE3SGPHSD7tATekZtRNEEvaMP9OlUnAvn2rn9aXVK85kjtBTO/TdHtsi8</latexit> xt <latexit sha1_base64="/9eKoYxPE1ZizRUsfEHodALxXRM=">AAACg3icdVHLSgMxFE1H66O+Wl26CZaCIJSZWtCNILpxqdAXtGPJpJk2mEyG5I60DPMfbvWv/BsztYs+9MKFw7mvw7lBLLgB1/0uOFvbxZ3dvf3SweHR8Um5ctoxKtGUtakSSvcCYpjgEWsDB8F6sWZEBoJ1g7fHvN59Z9pwFbVgFjNfknHEQ04JWOp1IAlMgjCdZsMUsmG56tbdeeBN4C1AFS3ieVgpDAcjRRPJIqCCGNP33Bj8lGjgVLCsNEgMiwl9I2PWtzAikhk/ncvOcM0yIxwqbTMCPGeXJ1IijZnJwHbmMs16LSf/qvUTCG/9lEdxAiyiv4fCRGBQOPcAj7hmFMTMAkI1t1oxnRBNKFinVja1PD/NxeVrVs4HMivV8DKTy4hBTlf7iBgre2Ei/6E53RyIDUusq2pkDbQv8dYfsAk6jbp3XW+8NKv3D4vn7KFzdIEukYdu0D16Qs+ojSjS6AN9oi+n6Fw5Daf52+oUFjNnaCWcux+lOcgZ</latexit> (b) Figure 5.2: Model Diagrams. (a) An autoregressive flow pre-processes a data sequence, x1:𝑇 , to produce a new sequence, y1:𝑇 , with reduced temporal dependencies. This simplifies dynamics modeling for a higher-level sequential latent variable model, 𝑝 𝜃 (y1:𝑇 , z1:𝑇 ). Empty diamond nodes represent deterministic dependencies, not recurrent states. (b) Diagram of the autoregressive flow architecture. Blank white rectangles represent convolutional layers (see Appendix). The three stacks of convolutional layers within the blue region are shared. cat denotes channel-wise concatenation. Data Dynamics The form of an affine autoregressive flow across sequences is given by x𝑡 = µ𝜃 (x<𝑡 ) + σ𝜃 (x<𝑡 ) and its inverse y𝑡 = y𝑡 , x𝑡 − µ𝜃 (x<𝑡 ) , σ𝜃 (x<𝑡 ) (5.2) (5.3) which, again, are equivalent to a Gaussian autoregressive model when y𝑡 ∼ N (y𝑡 ; 0, I). We can stack hierarchical chains of flows to improve the model capacity. Denoting 𝑚 the shift and scale functions at the 𝑚 th transform as µ𝑚 𝜃 (·) and σ𝜃 (·) respectively, we then calculate y𝑚 using the inverse transform: y𝑡𝑚 = 𝑚−1 y𝑡𝑚−1 − µ𝑚 𝜃 (y<𝑡 ) . σ𝜃𝑚 (y𝑚−1 <𝑡 ) (5.4) After the final (𝑀 th ) transform, we can choose the form of the base distribution, 𝑀 ), e.g., Gaussian. While we could attempt to model x 𝑝 𝜃 (y1:𝑇 1:𝑇 completely using stacked autoregressive flows, these models are limited to affine element-wise transforms that maintain the data dimensionality. Due to this limited capacity, purely flow-based models often require many transforms to be effective (Kingma and Dhariwal, 2018). 80 Instead, we can model the base distribution using an expressive sequential latent variable model (SLVM), or, equivalently, we can augment the conditional likelihood of a SLVM using autoregressive flows (Fig. 5.2a). Following the motivation from Section 5.2, the flow can remove temporal dependencies, simplifying the modeling task for the SLVM. With a single flow, the joint probability is   −1 𝜕x1:𝑇 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) = 𝑝 𝜃 (y1:𝑇 , z1:𝑇 ) det , (5.5) 𝜕y1:𝑇 where the SLVM distribution is given by 𝑝 𝜃 (y1:𝑇 , z1:𝑇 ) = 𝑇 Ö 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) 𝑝 𝜃 (z𝑡 |y<𝑡 , z<𝑡 ). (5.6) 𝑡=1 If the SLVM is itself a flow-based model, we can use maximum log-likelihood training. If not, we can resort to variational inference (Chung et al., 2015; Fraccaro et al., 2016). We derive and discuss this procedure in Section 5.5. Latent Dynamics We can also consider simplifying latent dynamics modeling using autoregressive flows in the latent prior. This is relevant in hierarchical SLVMs, such as VideoFlow (Kumar et al., 2020), where each latent variable is modeled as a function of past and higher-level latent variables. Using z𝑡(ℓ) to denote the latent variable at the ℓ th level at time 𝑡, we can parameterize the prior as ! (ℓ) −1 𝜕z (ℓ) (>ℓ) (ℓ) (>ℓ) 𝑡 𝑝 𝜃 (z𝑡(ℓ) |z<𝑡 , z𝑡 ) = 𝑝 𝜃 (u𝑡(ℓ) |u<𝑡 , z𝑡 ) det , (5.7) 𝜕u𝑡(ℓ) where we convert z𝑡(ℓ) into u𝑡(ℓ) using the inverse transform (ℓ) (ℓ) ). ))/β𝜃 (z<𝑡 u𝑡(ℓ) = (z𝑡(ℓ) − α𝜃 (z<𝑡 As noted previously, VideoFlow uses a special case of this procedure, setting (ℓ) (ℓ) (ℓ) α𝜃 (z<𝑡 ) = z𝑡−1 and β𝜃 (z<𝑡 ) = 1. Generalizing this procedure further simplifies dynamics throughout the model. 5.3 Experiments We demonstrate and evaluate the proposed technique on three benchmark video datasets: Moving MNIST (Srivastava, Mansimov, and Salakhudinov, 2015), KTH Actions (Schuldt, Laptev, and Caputo, 2004), and BAIR Robot Pushing (Ebert 81 x 1.0 0.5 Data Flow <latexit sha1_base64="pSMj62eOFVD7Buj853FHlma8izA=">AAACgnicdVHLSgNBEJys7/jWo5fBIIiHsKuCHjyIevCoYFRIltA76SSDMzvrTK8YlnyHV/0s/8bZmIMx2tBQVL+K6iRT0lEYflaCmdm5+YXFperyyura+sbm1r0zuRXYEEYZ+5iAQyVTbJAkhY+ZRdCJwofk6bKsP7ygddKkdzTIMNbQS2VXCiBPxS3CV7K6uAKCYXujFtbDUfBpEI1BjY3jpr1Zabc6RuQaUxIKnGtGYUZxAZakUDistnKHGYgn6GHTwxQ0urgYqR7yPc90eNdYnynxEftzogDt3EAnvlMD9d3vWkn+VWvm1D2NC5lmOWEqvg91c8XJ8NIC3pEWBamBByCs9Fq56IMFQd6oiU13UVyU4so1E+cTPazu8Z9MKSMj/TrZB6pn/IW+/oeWYnogc5h7V03HG+hfEv1+wDS4P6xHR/XD2+Pa+cX4OYtsh+2yfRaxE3bOrtkNazDBntkbe2cfwWxwEETB0XdrUBnPbLOJCM6+AEH1x3g=</latexit> <latexit sha1_base64="9c2S84X+Yjx7lgHsBqZAnAQM7c8=">AAACgnicdVHLSgMxFE3Hd321unQTLIK4KDNW0IWLoiAuK1gV2qFk0ts2mEzG5I62DP0Ot/pZ/o2Ztgtr9cKFw7mvw7lRIoVF3/8qeEvLK6tr6xvFza3tnd1See/B6tRwaHIttXmKmAUpYmiiQAlPiQGmIgmP0fN1Xn98BWOFju9xlECoWD8WPcEZOipsIwzRqOxG6rdxp1Txq/4k6CIIZqBCZtHolAuddlfzVEGMXDJrW4GfYJgxg4JLGBfbqYWE8WfWh5aDMVNgw2yiekyPHNOlPW1cxkgn7M+JjClrRypynYrhwP6u5eRftVaKvYswE3GSIsR8eqiXSoqa5hbQrjDAUY4cYNwIp5XyATOMozNqbtN9EGa5uHzN3PlIjYtH9CeTy0hQDef7mOxrd2Gg/qEFXxxILKTOVd11BrqXBL8fsAgeTqtBrXp6d1apX82es04OyCE5JgE5J3VySxqkSTh5Ie/kg3x6y96JF3i1aatXmM3sk7nwLr8BgEbHlg==</latexit> 0.0 0 1 2 3 4 5 6 7 8 9 Frame Number 10 11 12 13 14 15 16 3 4 5 6 7 8 9 Frame Number 10 11 12 13 14 15 16 y 0 Noise Base Dist. <latexit sha1_base64="xIsEvyuYXfmc5vw6eWNB2fL16SY=">AAACg3icdVHLSgMxFE3Hd33r0k2wCIJQZmpBN0LRjStR6ENox5JJb9tgMhmSO9Iy9D/c6l/5N2ZqF7bVCxcO574O50aJFBZ9/6vgrayurW9sbhW3d3b39g8Oj5pWp4ZDg2upzXPELEgRQwMFSnhODDAVSWhFr3d5vfUGxgod13GcQKjYIBZ9wRk66qWDMEKjsgctLEy6ByW/7E+DLoNgBkpkFo/dw0K309M8VRAjl8zaduAnGGbMoOASJsVOaiFh/JUNoO1gzBTYMJvKntAzx/RoXxuXMdIp+3siY8rasYpcp2I4tIu1nPyr1k6xfx1mIk5ShJj/HOqnkqKmuQe0JwxwlGMHGDfCaaV8yAzj6Jya21QPwiwXl6+ZOx+pSfGM/mZyGQmq0XwfkwPtLgzVP7TgywOJhdS5qnvOQPeSYPEBy6BZKQeX5cpTtVS7nT1nk5yQU3JOAnJFauSePJIG4cSQd/JBPr0178KreNWfVq8wmzkmc+HdfAN+ncgG</latexit> °10 <latexit sha1_base64="DGyz+2dcpzNiM3dz1AqEf/YJo60=">AAACinicdVFNSwMxEE3X71q12qOXYCl4KrtVUPEi6sGjglWhXUo2nbbBZLMks9Ky9Ld41Z/kvzHb9uBaHRh4vPl4j5kokcKi73+VvJXVtfWNza3ydmVnd6+6f/BkdWo4tLmW2rxEzIIUMbRRoISXxABTkYTn6PUmrz+/gbFCx484SSBUbBiLgeAMHdWr1roIYzQqu3ZL6K0TbE571brf9GdBl0GwAHWyiPvefqnX7WueKoiRS2ZtJ/ATDDNmUHAJ03I3tZAw/sqG0HEwZgpsmM3cT2nDMX060MZljHTG/pzImLJ2oiLXqRiO7O9aTv5V66Q4OA8zEScpQsznQoNUUtQ0PwXtCwMc5cQBxo1wXikfMcM4uoMVNj0GYZaby9cU5CM1LTfoTya3kaAaF/uYHGqnMFL/0IIvDyQWUndV3XcHdC8Jfj9gGTy1msFJs/VwWr+6XjxnkxySI3JMAnJGrsgduSdtwsmEvJMP8ulVvJZ34V3OW73SYqZGCuHdfgO42MnI</latexit> 0 1 2 Figure 5.3: Sequence Modeling with Autoregressive Flows. Top: Pixel values (solid) for a particular pixel location in a video sequence. An autoregressive flow models the pixel sequence using an affine shift (dashed) and scale (shaded), acting as a frame of reference. Middle: Frames of the data sequence (top) and the resulting “noise” (bottom) from applying the shift and scale. The redundant, static background has been largely removed. Bottom: The noise values (solid) are modeled using a base distribution (dashed and shaded) provided by a higher-level model. By removing temporal redundancy from the data sequence, the autoregressive flow simplifies dynamics modeling. et al., 2017). Experimental setups are first described, followed by a set of empirical analyses. Further details and additional results can be found in Marino, L. Chen, et al., 2020. Experimental Setup For data space modeling, we compare four model classes: 1) standalone affine autoregressive flows with one (1-AF) and 2) two (2-AF) transforms, 3) a sequential latent variable model (SLVM), and 4) SLVM with flow-based pre-processing (SLVM + AF). As we are not proposing a specific architecture, but rather a general modeling technique, the SLVM architecture is representative of recurrent convolutional video models with a single latent level (Denton and Fergus, 2018; Ha and Schmidhuber, 2018; Hafner et al., 2019). Flows are implemented with convolutional networks, taking in a fixed window of previous frames (Fig. 5.2b). These models allow us to evaluate the benefits of temporal pre-processing (SLVM vs. SLVM + AF) and the benefits of more expressive higher-level dynamics models (2-AF vs. SLVM + 1-AF). To evaluate latent dynamics modeling with autoregressive flows, we use the VISUALS: MOVING MNIST 82 x Flow µ✓ ✓ y yt = xt µ✓ (x<t ) ✓ (x<t ) Flow + Latent x Figure 5.4: Flow Visualization for SLVM + 1-AF on Moving MNIST (left) and µ✓ Actions (right). KTH ✓ tensor2tensor library (Vaswani et al., 2018) to compare 1) VideoFlow1 and y 2) the same model with affine autoregressive flow latent dynamics (VideoFlow + 21 AF). VideoFlow is significantly larger (3× more parameters) than the one-level SLVM, allowing us to evaluate whether autoregressive flows are beneficial in this high-capacity regime. To enable a fairer comparison in our experiments, models with autoregressive flow dynamics have comparable or fewer parameters than baseline counterparts. We note that autoregressive dynamics adds only a constant computational cost per time-step, and this computation can be parallelized for training and evaluation. Full architecture, training, and analysis details can be found in Marino, L. Chen, et al., 2020. Finally, as noted by Kumar et al. (Kumar et al., 2020), many previous works do not train SLVMs with proper log-likelihood objectives. Our SLVM results are consistent with previously reported log-likelihood values (Marino, Cvitkovic, and Yue, 2018) for the Stochastic Video Generation model (Denton and Fergus, 2018) trained with a log-likelihood bound objective. Analyses Visualization In Figure 5.3, we visualize the pre-processing procedure for SLVM + 1-AF on BAIR Robot Pushing. The plots show the RGB values for a pixel before (top) and after (bottom) the transform. The noise sequence is nearly zero throughout, despite large changes in the pixel value. We also see that the noise sequence (center, lower) is invariant to the static background, capturing the moving robotic arm. At some time steps (e.g., fourth frame), the autoregressive flow incorrectly predicts the next frame, however, the higher-level SLVM compensates for this prediction error. We also visualize each component of the flow. Figure 5.2b illustrates this for SLVM 1 We used a smaller version of the original model architecture, with half of the flow depth, due to GPU memory constraints. 83 (a) VideoFlow (b) VideoFlow + AF Figure 5.5: Improved Generated Samples. Random samples generated from (a) VideoFlow and (b) VideoFlow + AF, each conditioned on the first 3 frames. Using AF produces more coherent samples. The robot arm blurs for VideoFlow in samples 1 and 4 (red), but does not blur for VideoFlow + AF. + 1-AF on an input from BAIR Robot Pushing. We see that µ𝜃 captures the static background, while σ𝜃 highlights regions of uncertainty. In Figure 5.4 and the Appendix, we present visualizations on full sequences, where we see that different models remove varying degrees of temporal structure. Temporal Redundancy Reduction To quantify temporal redundancy reduction, we estimate the empirical correlation (linear dependence) between frames, denoted as corr, for the data and noise variables. This is an average normalized version of the auto-covariance of each signal with a time delay of 1 time step. Specifically, we estimate the temporal correlation as 𝐻,𝑊,𝐶 Õ   1 corrx ≡ · E𝑥 (𝑖, 𝑗,𝑘) ,𝑥 (𝑖, 𝑗,𝑘) ∼D 𝜉𝑡,𝑡+1 (𝑖, 𝑗, 𝑘) , 𝑡 𝐻𝑊𝐶 𝑖, 𝑗,𝑘 𝑡+1 (5.8) 84 1.0 corrx corry SLVM + 1-AF corry 1-AF 0.5 0.0 M-MNIST BAIR corry Correlation 1.0 0.5 0.0 KTH 0 50,000 Training Iterations (a) (b) Figure 5.6: Decreased Temporal Correlation. (a) Affine autoregressive flows result in sequences, y1:𝑇 , with decreased temporal correlation, corry , as compared with that of the original data, corrx . (b) For SLVM + 1-AF, corry decreases during training on KTH Actions. where the term inside the expectation is (𝑖, 𝑗,𝑘) 𝜉𝑡,𝑡+1 (𝑖, 𝑗, 𝑘) ≡ (𝑥𝑡 (𝑖, 𝑗,𝑘) − 𝜇 (𝑖, 𝑗,𝑘) )(𝑥𝑡+1
2 𝜎 (𝑖, 𝑗,𝑘) − 𝜇 (𝑖, 𝑗,𝑘) ) . (5.9) (𝑖, 𝑗,𝑘) Here, 𝑥𝑡 denotes the image at location (𝑖, 𝑗) and channel 𝑘 at time 𝑡, 𝜇 (𝑖, 𝑗,𝑘) is the empirical mean of this dimension, and 𝜎 (𝑖, 𝑗,𝑘) is the empirical standard deviation. 𝐻, 𝑊, and 𝐶 respectively denote the height, width, and number of channels of the observations, and D denotes the dataset. We define an analogous expression for y, denoted as corry . We evaluate corrx and corry for SLVM + 1-AF and 1-AF, with results shown in Figure 5.6a. In Figure 5.6b, we plot corry for SLVM + 1-AF during training on KTH Actions. Flows decrease temporal correlation, and base distributions without temporal structure (1-AF) yield comparatively more decorrelation. Temporal redundancy is progressively removed throughout training. However, due to the limited capacity of the flows, they are not capable of completely removing temporal correlations on the more complex datasets, necessitating the use of higher-level dynamics models. Performance Comparison Table 5.1 reports average negative log-likelihood results. Standalone flow-based models perform surprisingly well. Increasing flow depth from AF-1 to AF-2 generally results in improvement. SLVM + 1-AF outperforms the baseline SLVM despite having fewer parameters. Incorporating autoregressive flows into VideoFlow results in a modest but noticeable improvement, demonstrating that removing spatial dependencies, through VideoFlow, and temporal dependencies, through autoregressive flows, are complementary techniques. 85 Table 5.1: Quantitative Comparison. Average test and (train) negative loglikelihood in nats per dimension for Moving MNIST, BAIR Robot Pushing, and KTH Actions. Lower values are better. 1.5 BAIR KTH 2.15 (2.06) 2.13 (2.04) ≤ 1.92 (≤ 1.93) ≤ 1.86 (≤ 1.85) – – 3.05 (2.98) 2.90 (2.76) ≤ 3.57 (≤ 3.46) ≤ 2.35 (≤ 2.31) 1.53 (1.50) 1.50 (1.49) 3.34 (2.95) 3.35 (2.95) ≤ 4.63 (≤ 3.05) ≤ 2.39 (≤ 2.21) – – Train Test 1.0 1.0 0.5 0.0 Train Test 1.5 0.5 5 10 NLL Bound (a) SLVM 15 20 0.0 2 4 6 NLL Bound (b) SLVM + 1-AF 8 NLL Bound Gen. Gap 1-AF 2-AF SLVM SLVM + 1-AF VideoFlow VideoFlow + AF M-MNIST 1.5 1.0 SLVM + AF SLVM 0.5 0.5 Train Data Ratio 1.0 (c) Figure 5.7: Improved Generalization. The low-level reference frame improves generalization to unseen sequences. Train and test negative log-likelihood bound histograms for (a) SLVM and (b) SLVM + 1-AF on KTH Actions. (c) The generalization gap for SLVM + 1-AF remains small for varying amounts of KTH training data, while it becomes worse in the low-data regime for SLVM. Improved Samples The quantitative improvement over VideoFlow is less dramatic, as this is already a high-capacity model. However, qualitatively, we observe that incorporating autoregressive flow dynamics improves sample quality (Figure 5.5). In these randomly selected samples, the robot arm occasionally becomes blurry for VideoFlow (red boxes) but remains clear for VideoFlow + AF. Improved Generalization Our temporal normalization technique also improves generalization to unseen examples, a key benefit of normalization schemes, e.g., batch norm (Ioffe and Szegedy, 2015). Intuitively, higher-level dynamics are often preserved, whereas lower-level appearance is not. This is apparent on KTH Actions, which contains a substantial degree of train-test mismatch (different identities and activities). NLL histograms on KTH are shown in Figure 5.7, with greater overlap for SLVM + 1-AF. We also train SLVM and SLVM + 1-AF on subsets of the KTH Actions dataset. In Figure 5.7c, we see that autoregressive flows enable generalization 86 even in the low-data regime, whereas SLVM becomes worse. 5.4 Discussion This chapter introduced a technique for improving sequence modeling using autoregressive flows, yielding a learned form of feedforward processing. Learning a frame of reference, parameterized by autoregressive transforms, reduces temporal redundancy in input sequences, thereby simplifying downstream dynamics estimation. Thus, rather than expanding the model, we can simplify the input to meet the capacity of the model. We have analyzed and empirically shown how autoregressive pre-processing in both the data and latent spaces can improve sequence modeling performance and lead to improved sample quality and generalization. This approach is distinct from previous works with normalizing flows on sequences, yet contains connections to classical modeling and compression. Indeed, video compression schemes use (“inter-frame”) predictions as a frame of reference to remove temporal redundancy, enabling more efficient encoding (Oliver, 1952; Wiegand et al., 2003). Inspired by these techniques, follow-up work by Yang et al., 2021 applied sequential autoregressive flows to high-resolution video compression, demonstrating that this not only generalizes existing methods, but also yields improved compression performance. 5.5 Appendix: ELBO Derivation Consider the model defined in Section 3.3, with the conditional likelihood parameterized with autoregressive flows. That is, we parameterize x𝑡 = µ𝜃 (x<𝑡 ) + σ𝜃 (x<𝑡 ) (5.10) y𝑡 , yielding  𝜕x𝑡 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) = 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) det 𝜕y𝑡  −1 . (5.11) The joint distribution over all time steps is then given as 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) = 𝑇 Ö 𝑡=1 𝑇 Ö 𝑝 𝜃 (x𝑡 |x<𝑡 , z≤𝑡 ) 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 )  𝜕x𝑡 = 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) det 𝜕y𝑡 𝑡=1 (5.12)  −1 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ). (5.13) 87 To perform variational inference, we consider a filtering approximate posterior of the form 𝑇 Ö 𝑞(z1:𝑇 |x1:𝑇 ) = 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ). (5.14) 𝑡=1 We can then plug these expressions into the evidence lower bound: L ≡ E𝑞(z1:𝑇 |x1:𝑇 ) [log 𝑝 𝜃 (x1:𝑇 , z1:𝑇 ) − log 𝑞(z1:𝑇 |x1:𝑇 )] " !   −1 𝑇 Ö 𝜕x𝑡 = E𝑞(z1:𝑇 |x1:𝑇 ) log 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) det 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ) 𝜕y 𝑡 𝑡=1 !# 𝑇 Ö 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) − log (5.15) (5.16) 𝑡=1 = E𝑞(z1:𝑇 |x1:𝑇 )   # 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) 𝜕x𝑡 . log 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) − log − log det 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ) 𝜕y𝑡 𝑡=1 " 𝑇 Õ (5.17) Finally, in the filtering setting, we can rewrite the expectation, bringing it inside of the sum (see Gemici et al., 2017; Marino, Cvitkovic, and Yue, 2018): "   # 𝑇 Õ 𝑞(z𝑡 |x≤𝑡 , z<𝑡 ) 𝜕x𝑡 . L= E𝑞(z ≤𝑡 |x ≤𝑡 ) log 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) − log − log det 𝑝 𝜃 (z𝑡 |x<𝑡 , z<𝑡 ) 𝜕y𝑡 𝑡=1 (5.18) Because there exists a one-to-one mapping between x1:𝑇 and y1:𝑇 , we can equivalently condition the approximate posterior and the prior on y, i.e. "   # 𝑇 Õ 𝜕x𝑡 𝑞(z𝑡 |y≤𝑡 , z<𝑡 ) . − log det L= E𝑞(z ≤𝑡 |y ≤𝑡 ) log 𝑝 𝜃 (y𝑡 |y<𝑡 , z≤𝑡 ) − log 𝑝 (z |y , z ) 𝜕y 𝜃 𝑡 <𝑡 <𝑡 𝑡 𝑡=1 (5.19) References Agrawal, Siddharth and Ambedkar Dukkipati (2016). “Deep Variational Inference Without Pixel-Wise Reconstruction”. In: arXiv preprint arXiv:1611.05209. Barlow, Horace B et al. (1961). “Possible principles underlying the transformation of sensory messages”. In: Sensory communication 1, pp. 217–234. Chen, Scott Saobing and Ramesh A Gopinath (2001). “Gaussianization”. In: Advances in neural information processing systems, pp. 423–429. Chua, Kurtland et al. (2018). “Deep reinforcement learning in a handful of trials using probabilistic dynamics models”. In: Advances in Neural Information Processing Systems, pp. 4754–4765. 88 Chung, Junyoung et al. (2015). “A recurrent latent variable model for sequential data”. In: Advances in neural information processing systems, pp. 2980–2988. Deco, Gustavo and Wilfried Brauer (1995). “Higher order statistical decorrelation without information loss”. In: Advances in Neural Information Processing Systems, pp. 247–254. Denton, Emily and Rob Fergus (2018). “Stochastic Video Generation with a Learned Prior”. In: International Conference on Machine Learning, pp. 1182–1191. Dinh, Laurent, David Krueger, and Yoshua Bengio (2015). “Nice: Non-linear independent components estimation”. In: International Conference on Learning Representations. Ebert, Frederik et al. (2017). “Self-Supervised Visual Planning with Temporal Skip Connections”. In: Conference on Robot Learning. Fraccaro, Marco et al. (2016). “Sequential neural models with stochastic layers”. In: Advances in neural information processing systems, pp. 2199–2207. Friedman, Jerome H (1987). “Exploratory projection pursuit”. In: Journal of the American statistical association 82.397, pp. 249–266. Friston, Karl (2008). “Hierarchical models in the brain”. In: PLoS computational biology 4.11, e1000211. Gemici, Mevlana et al. (2017). “Generative Temporal Models with Memory”. In: arXiv preprint arXiv:1702.04649. Ha, David and Jürgen Schmidhuber (2018). “Recurrent world models facilitate policy evolution”. In: Advances in Neural Information Processing Systems, pp. 2450– 2462. Hafner, Danijar et al. (2019). “Learning Latent Dynamics for Planning from Pixels”. In: International Conference on Machine Learning, pp. 2555–2565. Huang, Chin-Wei et al. (2017). “Learnable explicit density for continuous latent space and variational inference”. In: arXiv preprint arXiv:1710.02248. Hyvärinen, Aapo and Erkki Oja (2000). “Independent component analysis: algorithms and applications”. In: Neural networks 13.4-5, pp. 411–430. Ioffe, Sergey and Christian Szegedy (2015). “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: International Conference on Machine Learning, pp. 448–456. Kingma, Durk P and Prafulla Dhariwal (2018). “Glow: Generative flow with invertible 1x1 convolutions”. In: Advances in Neural Information Processing Systems, pp. 10215–10224. Kingma, Durk P, Tim Salimans, et al. (2016). “Improved variational inference with inverse autoregressive flow”. In: Advances in neural information processing systems, pp. 4743–4751. 89 Kumar, Manoj et al. (2020). “VideoFlow: A Flow-Based Generative Model for Video”. In: International Conference on Learning Representations. Laparra, Valero, Gustavo Camps-Valls, and Jesús Malo (2011). “Iterative gaussianization: from ICA to random rotations”. In: IEEE transactions on neural networks 22.4, pp. 537–549. Marino, Joseph, Lei Chen, Jiawei He, and Stephan Mandt (2020). “Improving Sequential Latent Variable Models with Autoregressive Flows”. In: Symposium on Advances in Approximate Bayesian Inference, pp. 1–16. url: http : / / proceedings.mlr.press/v118/marino20a.html. Marino, Joseph, Milan Cvitkovic, and Yisong Yue (2018). “A general method for amortizing variational filtering”. In: Advances in Neural Information Processing Systems, pp. 7857–7868. url: http://papers.nips.cc/paper/8011- ageneral-method-for-amortizing-variational-filtering. Oliver, BM (1952). “Efficient coding”. In: The Bell System Technical Journal 31.4, pp. 724–750. Pourahmadi, Mohsen (2011). “Covariance estimation: The GLM and regularization perspectives”. In: Statistical Science, pp. 369–387. Schuldt, Christian, Ivan Laptev, and Barbara Caputo (2004). “Recognizing human actions: a local SVM approach”. In: International Conference on Pattern Recognition. Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov (2015). “Unsupervised learning of video representations using lstms”. In: International conference on machine learning, pp. 843–852. Vaswani, Ashish et al. (2018). “Tensor2Tensor for Neural Machine Translation”. In: CoRR abs/1803.07416. url: http://arxiv.org/abs/1803.07416. Wiegand, Thomas et al. (2003). “Overview of the H. 264/AVC video coding standard”. In: IEEE Transactions on circuits and systems for video technology 13.7, pp. 560– 576. Winkler, Christina et al. (2019). “Learning Likelihoods with Conditional Normalizing Flows”. In: arXiv preprint arXiv:1912.00042. Yang, Ruihan, Yibo Yang, Joseph Marino, and Stephan Mandt (2021). “Hierarchical Autoregressive Modeling for Neural Video Compression”. In: International Conference on Learning Representations. url: https://openreview.net/ pdf?id=TK_6nNb_C7q. Part III Control 90 91 Chapter 6 ITERATIVE AMORTIZED POLICY OPTIMIZATION learned negative feedback control 14 Marino, Joseph, Alexandre Piché, and Yisong Yue (2019). “On the Design of Variational RL Algorithms”. In: NeurIPS Workshop on Deep Reinforcement Learning. url: https://drive.google.com/file/d/10hB0AS_naGNgSNG8p1kqElnc9HRmMYde/ view?usp=drivesdk. Marino, Joseph, Alexandre Piché, Alessandro Davide Ialongo, and Yisong Yue (2020). “Iterative Amortized Policy Optimization”. In: Preprint. url: https: //arxiv.org/abs/2010.10670. 6.1 Introduction Chapters 3 and 4 investigated improved methods for perceptual inference using learned negative feedback. In this chapter, we apply the same approach to control. On the surface, perception and control may appear to be disparate settings; perception, as we have framed it, involves an internal generative model, whereas control is typically formulated with an agent in a pre-defined environment along with a reward (or cost) function. Yet, by specifying the probabilistic computation graph implied by the agent-environment interaction, and with an appropriate mathematical framing, we can apply the techniques of probabilistic graphical models and variational inference to control (Levine, 2018). While the notion of framing control in terms of variational calculus dates back to cybernetics (Wiener, 1948), it is only recently that this connection with probabilistic inference has garnered renewed attention in the control 92 and reinforcement learning communities (Toussaint and Storkey, 2006; Todorov, 2008; Levine, 2018). In general, reinforcement learning (RL) algorithms involve policy evaluation and policy optimization (Sutton and Barto, 2018). Given a policy, one can estimate the value for each state or state-action pair following that policy, and given a value estimate, one can improve the policy to maximize the value. This latter procedure, policy optimization, can be challenging in continuous control due to instability and poor asymptotic performance. In deep RL, where policies over continuous actions are often parameterized by deep networks, such issues are typically tackled using regularization from previous policies (Schulman, Levine, et al., 2015; Schulman, Wolski, et al., 2017) or by maximizing policy entropy (V. Mnih, Badia, et al., 2016; Fox, Pakman, and Tishby, 2016). These techniques can be interpreted as variational inference (Levine, 2018), using optimization to infer a policy that yields high expected return while satisfying prior policy constraints. This smooths the optimization landscape, improving stability and performance (Ahmed et al., 2019). However, one subtlety arises: when used with entropy or KL regularization, policy networks perform amortized optimization (Gershman and Goodman, 2014). That is, rather than optimizing the action distribution, e.g., mean and variance, many deep RL algorithms, such as soft actor-critic (SAC) (Haarnoja, Zhou, Abbeel, et al., 2018; Haarnoja, Zhou, Hartikainen, et al., 2018), instead optimize a network to output these parameters, learning to optimize the policy. Typically, this is implemented as a direct mapping from states to action distribution parameters. While such direct amortization schemes have improved the efficiency of variational inference as “encoder” networks (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014; A. Mnih and Gregor, 2014), they also suffer from several drawbacks: 1) they tend to provide suboptimal estimates (Cremer, Li, and Duvenaud, 2018; Kim et al., 2018; Marino, Yue, and Mandt, 2018), yielding a so-called “amortization gap” in performance (Cremer, Li, and Duvenaud, 2018), 2) they are restricted to a single estimate (Greff et al., 2019), thereby limiting exploration, and 3) they cannot generalize to new objectives, unlike, e.g., gradient-based (Henaff, Whitney, and LeCun, 2017) or gradient-free optimizers (Rubinstein and Kroese, 2013). Inspired by techniques and improvements from variational inference, we investigate iterative amortized policy optimization. Iterative amortization (Chapter 3) uses gradients or errors to iteratively update the parameters of a distribution. Unlike direct amortization, which receives gradients only after outputting the distribution, iterative 93 amortization uses these gradients online, thereby learning to iteratively optimize. In generative modeling settings, iterative amortization empirically outperforms direct amortization (Chapters 3 & 4) and can find multiple modes of the optimization landscape (Greff et al., 2019). We evaluate iterative amortized policy optimization on the suite of OpenAI gym MuJoCo environments (Todorov, Erez, and Tassa, 2012; Brockman et al., 2016), with performance improvements over direct amortized policies, as well as more complex flow-based policies. We also demonstrate novel benefits of this amortization technique: improved accuracy, providing multiple policy estimates, and generalizing to new objectives. 6.2 Background Preliminaries We consider Markov decision processes (MDPs), where s𝑡 ∈ S and a𝑡 ∈ A are the state and action at time 𝑡, resulting in reward 𝑟 𝑡 = 𝑟 (s𝑡 , a𝑡 ). Environment state transitions are given by s𝑡+1 ∼ 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ), and the agent is defined by a parametric distribution, 𝑝 𝜃 (a𝑡 |s𝑡 ), with parameters 𝜃. The discounted sum of Í rewards is denoted as R (𝜏) = 𝑡 𝛾 𝑡 𝑟 𝑡 , where 𝛾 ∈ (0, 1] is the discount factor, and 𝜏 = (s1 , a1 , . . . ) is a trajectory. The distribution over trajectories is 𝑝(𝜏) = 𝜌(s1 ) 𝑇 Ö 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ) 𝑝 𝜃 (a𝑡 |s𝑡 ), (6.1) 𝑡=1 where the initial state is drawn from the distribution 𝜌(s1 ). The standard RL objective consists of maximizing the expected discounted return, E 𝑝(𝜏) [R (𝜏)]. For convenience of presentation, we use the undiscounted setting (𝛾 = 1), though the formulation can be applied with any valid 𝛾. KL-Regularized Reinforcement Learning Various works have formulated RL, planning, and control problems in terms of probabilistic inference (Dayan and Hinton, 1997; Attias, 2003; Verma and Rao, 2006; Toussaint and Storkey, 2006; Todorov, 2008; Botvinick and Toussaint, 2012; Levine, 2018). These approaches consider the agent-environment interaction as a graphical model, then convert reward maximization into maximum marginal likelihood estimation, learning and inferring a policy that results in maximal reward. This conversion is accomplished by introducing one or more binary observed variables (Cooper, 1988), denoted as O, with
𝑝(O = 1|𝜏) ∝ exp R (𝜏)/𝛼 , 94 where 𝛼 is a temperature hyper-parameter. These new variables are often referred to as “optimality” variables (Levine, 2018). We would like to infer latent variables, 𝜏, and learn parameters, 𝜃, that yield the maximum log-likelihood of optimality, i.e., log 𝑝(O = 1). Evaluating this likelihood requires marginalizing the joint distribution, ∫ 𝑝(O = 1) = 𝑝(𝜏, O = 1)𝑑𝜏. This involves averaging over all trajectories, which is intractable in high-dimensional spaces. Instead, we can use variational inference to lower bound this objective, introducing a structured approximate posterior distribution: 𝜋(𝜏|O) = 𝑇 Ö 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 )𝜋(a𝑡 |s𝑡 , O). (6.2) 𝑡=1 This provides the following lower bound on the objective, log 𝑝(O = 1):   ∫ ∫ 𝑝(𝜏) log 𝑝(O = 1|𝜏) 𝑝(𝜏)𝑑𝜏 ≥ 𝜋(𝜏|O) log 𝑝(O = 1|𝜏) + log 𝑑𝜏 (6.3) 𝜋(𝜏|O) = E𝜋 [R (𝜏)/𝛼] − 𝐷 KL (𝜋(𝜏|O)k 𝑝(𝜏)). (6.4) Equivalently, we can multiply by 𝛼, defining the variational RL objective as J (𝜋, 𝜃) ≡ E𝜋 [R (𝜏)] − 𝛼𝐷 KL (𝜋(𝜏|O)k 𝑝(𝜏)). (6.5) This objective consists of the expected return (i.e., the standard RL objective) and a KL divergence between 𝜋(𝜏|O) and 𝑝(𝜏). In terms of states and actions, this objective is written as # " 𝑇 Õ 𝜋(a𝑡 |s𝑡 , O) J (𝜋, 𝜃) = Es𝑡 ,𝑟 𝑡 ∼𝑝env 𝑟 𝑡 − 𝛼 log . (6.6) a𝑡 ∼𝜋 𝑝 𝜃 (a𝑡 |s𝑡 ) 𝑡=1 At a given timestep, 𝑡, one can optimize this objective by estimating the future terms in the summation using a “soft” action-value (𝑄 𝜋 ) network (Haarnoja, H. Tang, et al., 2017) or model (Piché et al., 2019). For instance, sampling s𝑡 ∼ 𝑝 env , slightly abusing notation, we can write the objective at time 𝑡 as J (𝜋, 𝜃) = E𝜋 [𝑄 𝜋 (s𝑡 , a𝑡 )] − 𝛼𝐷 KL (𝜋(a𝑡 |s𝑡 , O)|| 𝑝 𝜃 (a𝑡 |s𝑡 )). (6.7) Policy optimization in the KL-regularized setting corresponds to maximizing J w.r.t. 𝜋. We often consider parametric policies, in which 𝜋 is defined by distribution parameters, λ, e.g., Gaussian mean, µ, and variance, σ 2 . In this case, policy optimization corresponds to maximizing: λ ← arg max J (𝜋, 𝜃). λ (6.8) Optionally, we can then also learn the policy prior parameters, 𝜃 (Abdolmaleki et al., 2018). 95 1.0 334 E⇡ [Q⇡ (st , at )] <latexit sha1_base64="zPQeLlZ86knaMERRK34nAbJ7OzI=">AAACt3icdVFda9swFFXcbm2zj6bd417EQqGDEex20O0ttAz22ELTFmJjZOU6FpUsTboeC8Z/p79mryvs30xOUmia9YLg3HO/ju7NjBQOw/BvJ9jYfPFya3un++r1m7e7vb39K6cry2HEtdT2JmMOpChhhAIl3BgLTGUSrrPbszZ+/ROsE7q8xJmBRLFpKXLBGXoq7Q1jxbDIsvpbk8ZG0FhCjmN6MXcOF8G8dk2Kn+iDx7z3kcZWTAtM0l4/HIRzo+sgWoI+Wdp5utdJ44nmlYISuWTOjaPQYFIzi4JLaLpx5cAwfsumMPawZApcUs+/2tADz0xorq1/JdI5+7iiZsq5mcp8ZqvWPY215P9i4wrzL0ktSlMhlHwxKK8kRU3bvdGJsMBRzjxg3AqvlfKCWcbRb3el02WU1K24ts3K+Ew13QP6mGllGFS/VvOYnGo/oVDP0IKvFxgHld+qnvgF+pNETw+wDq6OBtHx4Ojic394ujzONnlPPpBDEpETMiTfyTkZEU7uyG/yh9wHX4M0yINikRp0ljXvyIoFP/4BRpjcEQ==</latexit> Q⇡ (st , at ) <latexit sha1_base64="MJE81UhXK+zdK2r8FWvIKlHN2Uw=">AAACmXicdVHLSgMxFE3Hd31VXboJLYKClBkVdOljI65atFVohyGTZtpgMgnJHbEM3fs1bvVX/BsztYK1eiFw7rnPnBtrwS34/kfJm5tfWFxaXimvrq1vbFa2tttWZYayFlVCmYeYWCZ4ylrAQbAHbRiRsWD38eNVEb9/YsZyld7BULNQkn7KE04JOCqqVJtRV3O835UEBnGS21EEh/jbI847iCo1v+6PDc+CYAJqaGKNaKsUdXuKZpKlQAWxthP4GsKcGOBUsFG5m1mmCX0kfdZxMCWS2TAff2aE9xzTw4ky7qWAx+zPipxIa4cydpnFlvZ3rCD/inUySM7CnKc6A5bSr0FJJjAoXCiDe9wwCmLoAKGGu10xHRBDKDj9pjrdBWFeLFe0mRofy1F5D/9kijU0yOfpPCL6yk0YyH9oTmcLtGWZU1X1nIDuJMHvA8yC9lE9OK4fNU9q55eT4yyjXVRF+yhAp+gcXaMGaiGKXtArekPv3q534V17N1+pXmlSs4OmzLv9BH4Tz7E=</latexit> 333 0.0 J tanh(µ3) 0.5 <latexit sha1_base64="/MMpvXKoB+n+blZ12rYPUy/58KI=">AAADBXicfVHdatRAFJ7Ev7r+dKt3ejO4FlrQJamCXhZF8M4K3bawCeFkdpIMnckMmRNxGXPtnW/inXjrc/gePoCT3RW6XfXAMN/5zi/ny40UFqPoZxBeuXrt+o2tm4Nbt+/c3R7u3Duxum0YnzAtdXOWg+VS1HyCAiU/Mw0HlUt+mp+/7uOnH3hjha6PcW54qqCsRSEYoKey4ZenCUhTAU0UYJXn7k2XJUbQRPICp/7TJU2KBpjzrI9Ugu4tUwsHXYb0E/3j2t59snQZSPeu2++cyRKsOMJ/q/Y7mjSirDDNhqNoHC2MboJ4BUZkZUfZTpAlM81axWtkEqydxpHB1EGDgkneDZLWcgPsHEo+9bAGxW3qFofr6K5nZrTQjX810gV7scKBsnaucp/Zr2svx3ryb7Fpi8XL1InatMhrthxUtJKipr0KdCYazlDOPQDWCL8rZRX4M6PXaq3TcZy6frm+zdr4XHWDXXqR6dcwqD6u54EstZ9QqX/Qgm0WGMtbf1U98wf0ksSXBdgEJwfj+Nn44P3z0eGrlThb5CF5RPZITF6QQ/KWHJEJYeRX8CAYBY/Dz+HX8Fv4fZkaBqua+2TNwh+/AQJ1+Y4=</latexit> −0.5 −1.0 −1.0  ⇡ (at |st , O) ↵E⇡ log p✓ (at |st ) ⇡ (at |st , O) <latexit sha1_base64="cf2O3KIgEuC4a9BQLouEV1r8y6Q=">AAACqnicdVHdTtswFHbDNljHtsIud2OtQgIJVQlMgks0brgbmygwNVF04jithR1b9glaFfIge5rdskfY28xpO4lSOJKl73zn7/M5mZHCYRj+7QRrL16+Wt943X2z+fbd+97W9qXTlWV8yLTU9joDx6Uo+RAFSn5tLAeVSX6V3Zy28atbbp3Q5QVODU8UjEtRCAboqbR3GBuRxmYi6G6sACdZUUOTIr2j/13n3f25x0DWX5u9tNcPB+HM6CqIFqBPFnaebnXSONesUrxEJsG5URQaTGqwKJjkTTeuHDfAbmDMRx6WoLhL6tnvGrrjmZwW2vpXIp2xDytqUM5NVeYzW5Hucawln4qNKiyOk1qUpkJesvmgopIUNW1XRXNhOUM59QCYFV4rZROwwNAvdKnTRZTUrbi2zdL4TDXdHfqQaWUYVD+X80COtZ8wUc/Qgq0WGMcrv1Wd+wX6k0SPD7AKLg8G0eHg4Nvn/smXxXE2yEfyieySiByRE3JGzsmQMPKL/Cb35E+wH3wPfgSjeWrQWdR8IEsW5P8AlJfW4A==</latexit> <latexit sha1_base64="GH6lmsh/3Bla21UIjWoARjx3LaU=">AAACnXicdVHdahNBFJ6s1db401QvvXAwVOpN2K1CvSxKoRdFKjRNIVmWs5OzydCZnWHmrBjWvIFP4219kb5NZ9MITVMPDHznO7/zndwq6SmOr1vRo43HTza3nrafPX/xcruz8+rcm8oJ7AujjLvIwaOSJfZJksIL6xB0rnCQX35t4oMf6Lw05RnNLKYaJqUspAAKVNZ5b7MRTZGA74000DQvaphnxH/xf64P7oes04178cL4OkiWoMuWdprttLLR2IhKY0lCgffDJLaU1uBICoXz9qjyaEFcwgSHAZag0af14kNzvhuYMS+MC68kvmDvVtSgvZ/pPGQ2W/r7sYZ8KDasqPic1rK0FWEpbgcVleJkeKMOH0uHgtQsABBOhl25mIIDQUHDlU5nSVo3yzVtVsbnet7e5XeZZg1L+udqHqiJCROm+j+0FOsF1mMVVDXjIGA4SXL/AOvgfL+XfOztf//UPfyyPM4We8PesT2WsAN2yI7ZKeszwX6zP+yK/Y3eRkfRSfTtNjVqLWtesxWLBjcLFNGl</latexit> 332 Direct Policy Network Iterative Policy Network Optimal Estimate −0.5 331 330 0.0 tanh(µ1) 0.5 1.0 (a) <latexit sha1_base64="XVEFz5wPUvNmOe8ykG++NxVNAiY=">AAACu3icdVHbbhMxEHWWWwm3FB55sYgqFQmi3YIEL0gV8MBjkZq2UrJaeZ3ZxK1vsmerRmZ/iK/htXwN3nSFkgZGsnR8ZsZzfKa0UnhM0+tecufuvfsPdh72Hz1+8vTZYPf5iTe14zDmRhp3VjIPUmgYo0AJZ9YBU6WE0/LiS5s/vQTnhdHHuLSQKzbXohKcYaSKwVdbhCnCFToVQF82Dd2fKoaLsgq+KZD+oGvXgG+z5s1fhnXM62IwTEfpKug2yDowJF0cFbu9YjozvFagkUvm/SRLLeaBORRcQtOf1h4s4xdsDpMINVPg87D6bkP3IjOjlXHxaKQrdr0jMOX9UpWxslXqb+da8l+5SY3VxzwIbWsEzW8GVbWkaGjrHZ0JBxzlMgLGnYhaKV8wxzhGhzdeOs7y0Iprn9kYX6qmv0fXmVaGRXW1Wcfk3MQJC/UfWvDtBuuhjq6aWTQwriS7vYBtcHIwyt6NDr6/Hx5+7pazQ16SV2SfZOQDOSTfyBEZE05+kl/kmvxOPiU8OU/kTWnS63pekI1I6j9rsd6c</latexit> Direct p✓ (at |st ) penv (st |st 1 , at 1 ) Iterative <latexit sha1_base64="NHTlNGKuf9jvAAkt9p+DKT273ZM=">AAAChnicdVHLTgIxFC3jC/EB6NJNIzFxRWZ8BJdEXbjERMAEJqRTLtDYTiftHQOZ8CVu9aP8GzvIQkRv0uTk3NfpuVEihUXf/yx4G5tb2zvF3dLe/sFhuVI96lidGg5trqU2zxGzIEUMbRQo4TkxwFQkoRu93OX57isYK3T8hLMEQsXGsRgJztBRg0q5jzBFo7J7YYDjfFCp+XV/EXQdBEtQI8toDaqFQX+oeaogRi6Ztb3ATzDMmEHBJcxL/dRCwvgLG0PPwZgpsGG2UD6nZ44Z0pE27sVIF+zPjowpa2cqcpWK4cT+zuXkX7leiqObMBNxkiLE/HvRKJUUNc1toMPFd+XMAcaNcFopnzDDODqzViY9BWGWi8vHrKyP1Lx0Rn8yuYwE1XS1jsmxdhsm6h9a8PWGxELqXNVDZ6A7SfD7AOugc1EPLusXj1e15u3yOEVyQk7JOQlIgzTJA2mRNuEkJW/knXx4Ra/uXXuN71KvsOw5JivhNb8A8zLIng==</latexit> <latexit sha1_base64="su8BrL+95YX6wSbmN0WLJ+hC4sU=">AAACiXicdVHLSgMxFE3Hd+uj6tJNsAiuyowKiivRje4U+hDaoWTS2zY0mQzJHWkZ5lfc6i/5N2ZqF7bVCxcO574O50aJFBZ9/6vkra1vbG5t75Qru3v7B9XDo5bVqeHQ5Fpq8xoxC1LE0ESBEl4TA0xFEtrR+KGot9/AWKHjBk4TCBUbxmIgOENH9apHXYQJGpU9IRjHvUHeq9b8uj8LugqCOaiReTz3Dku9bl/zVEGMXDJrO4GfYJgxg4JLyMvd1ELC+JgNoeNgzBTYMJuJz+mZY/p0oI3LGOmM/T2RMWXtVEWuUzEc2eVaQf5V66Q4uAkzEScpQsx/Dg1SSVHTwgnaFwY4yqkDjBvhtFI+YoZx58TipkYQZoW4Ys3C+Ujl5TP6mylkJKgmi31MDrW7MFL/0IKvDiQWUueq7jsD3UuC5QesgtZFPbisX7xc1e7u58/ZJifklJyTgFyTO/JInkmTcDIh7+SDfHoVL/BuvNufVq80nzkmC+E9fAM0u8oO</latexit> (b) 21 Figure 6.1: Amortized Policy Optimization. (a) 2D visualization of policy optimization. A direct amortized policy network fails to output an optimal estimate, resulting in an amortization gap in performance. An iterative amortized policy network finds an improved estimate. (b) Diagrams of direct and iterative amortized policy optimization. Larger circles denote distributions, and smaller red circles denote terms in the objective, J . Dashed arrows denote amortized optimizers. Iterative amortization uses gradient-based feedback during optimization, whereas direct amortization does not. Entropy & KL Regularized Policy Networks Perform Direct Amortization Policy-based approaches to RL typically do not directly optimize the action distribution parameters, e.g., through gradient-based optimization. Instead, the action distribution parameters are output by a function approximator (deep network), 𝑓𝜙 , which is trained using deterministic (Silver et al., 2014; Lillicrap et al., 2016) or stochastic gradients (Williams, 1992; Heess et al., 2015). When combined with entropy or KL regularization, this policy network is a form of amortized optimization (Gershman and Goodman, 2014), learning to estimate policies. Again, denoting the action distribution parameters, e.g., mean and variance, as λ, for a given state, s, we can express this direct mapping as λ ← 𝑓𝜙 (s), (direct amortization) (6.9) denoting the corresponding policy as 𝜋 𝜙 (a|s, O; λ). Thus, 𝑓𝜙 attempts to learn to optimize Eq. 6.8. This setup is shown in Figure 6.1 (Right). Without entropy or KL regularization, i.e., 𝜋 𝜙 (a|s) = 𝑝 𝜃 (a|s), we can instead interpret the network as directly integrating the LHS of Eq. 6.3, which is less efficient and more challenging. Regularization smooths the optimization landscape, yielding more stable improvement and higher asymptotic performance (Ahmed et al., 2019). Viewing policy networks as a form of direct amortized variational optimizer (Eq. 6.9) allows us to see that they are similar to “encoder” networks in variational autoencoders 96 Algorithm 3 Direct Amortization Initialize 𝜙 for each environment step do λ ← 𝑓𝜙 (s𝑡 ) a𝑡 ∼ 𝜋 𝜙 (a𝑡 |s𝑡 , O; λ) s𝑡+1 ∼ 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ) end for for each training step do 𝜙 ← 𝜙 + 𝜂∇𝜙 J end for Algorithm 4 Iterative Amortization Initialize 𝜙 for each environment step do Initialize λ for each policy optimization iteration do λ ← 𝑓𝜙 (s𝑡 , λ, ∇λ J ) end for a𝑡 ∼ 𝜋 𝜙 (a𝑡 |s𝑡 , O; λ) s𝑡+1 ∼ 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ) end for for each training step do 𝜙 ← 𝜙 + 𝜂∇𝜙 J end for (VAEs) (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). However, there are several drawbacks to direct amortization. Amortization Gap. Direct amortization results in suboptimal approximate posterior estimates, with the resulting gap in the variational bound referred to as the amortization gap (Cremer, Li, and Duvenaud, 2018). Thus, in the RL setting, an amortized policy, 𝜋 𝜙 , results in worse performance than the optimal policy within the parametric policy class, denoted as b 𝜋 . The amortization gap is the gap in following inequality: J (𝜋 𝜙 , 𝜃) ≤ J (b 𝜋 , 𝜃). Because J is a variational bound on the RL objective, i.e., expected return, a looser bound, due to amortization, prevents one from more completely optimizing this objective. This is shown in Figure 6.1 (Left),1 where J is plotted over two dimensions of the policy mean at a particular state in the MuJoCo environment Hopper-v2. The estimate of a direct amortized policy () is suboptimal, far from the optimal estimate (F). While the relative difference in the objective is relatively small, suboptimal estimates prevent sampling and exploring high-value regions of the action-space. That is, suboptimal estimates have only a minor impact on evaluation performance (see Appendix B.4) but hinder effective data collection. 1 Additional 2D plots are shown in Appendix Figure B.3. 97 Single Estimate. Direct amortization is limited to a single, static estimate. In other words, if there are multiple high-value regions of the action-space, a unimodal (e.g., Gaussian) direct amortized policy is restricted to only one region, thereby limiting exploration. Note that this is an additional restriction beyond simply considering uni-modal distributions, as a generic optimization procedure may arrive at multiple uni-modal estimates depending on initialization and stochastic sampling (see Section 6.3). While multi-modal distributions reduce the severity of this restriction (Y. Tang and Agrawal, 2018; Haarnoja, Hartikainen, et al., 2018), the other limitations of direct amortization still persist. Inability to Generalize Across Objectives. Direct amortization is a feedforward procedure, receiving gradients from the objective only after estimation. This is contrast to other forms of optimization, which receive gradients (feedback) during estimation. Thus, unlike other optimizers, direct amortization is incapable of generalizing to new objectives, e.g., if 𝑄 𝜋 (s, a) or 𝑝 𝜃 (a|s) change, which is a desirable capability for adapting to new tasks or environments. To improve upon this scheme and overcome these drawbacks, in Section 6.3, we turn to iterative amortization (Chapter 3), retaining the efficiency of amortization while employing a more flexible iterative estimation procedure. Related Work Previous works have investigated methods for improving policy optimization. QT-Opt (Kalashnikov et al., 2018) uses the cross-entropy method (CEM) (Rubinstein and Kroese, 2013), an iterative derivative-free optimizer, to optimize a 𝑄-value estimator for robotic grasping. CEM and related methods are also used in model-based RL for performing model-predictive control (Nagabandi et al., 2018; Chua et al., 2018; Piché et al., 2019; Hafner et al., 2019). Gradient-based policy optimization (Henaff, Whitney, and LeCun, 2017; Srinivas et al., 2018; Bharadhwaj, Xie, and Shkurti, 2020), in contrast, is less common, however, gradient-based optimization can also be combined with CEM (Amos and Yarats, 2020). Most policy-based methods use direct amortization, either using a feedforward (Haarnoja, Zhou, Abbeel, et al., 2018) or recurrent (Guez et al., 2019) network. Similar approaches have also been applied to model-based value estimates (Byravan et al., 2020; Clavera, Y. Fu, and Abbeel, 2020; Amos, Stanton, et al., 2020), as well as combining direct amortization with model predictive control (Lee, Saigol, and Theodorou, 2019) and planning (Rivière et al., 2020). A separate line of work has explored improving the policy distribution, 98 using normalizing flows (Haarnoja, Hartikainen, et al., 2018; Y. Tang and Agrawal, 2018) and latent variables (Tirumala et al., 2019). In principle, iterative amortization can perform policy optimization in each of these settings. Iterative amortized policy optimization is conceptually similar to negative feedback control (Astrom and Murray, 2008), using errors to update policy estimates. However, while conventional feedback control methods are often restricted in their applicability, e.g., linear systems and quadratic cost, iterative amortization is generally applicable to any differentiable control objective. This is analogous to the generalization of Kalman filtering (Kalman, 1960) to amortized variational filtering (Chapter 4) for state estimation. 6.3 Iterative Amortized Policy Optimization Formulation Iterative amortized optimizers (Chapter 3) utilize some form of error or gradient to update the approximate posterior distribution parameters. While various forms exist, we consider gradient-encoding models (Andrychowicz et al., 2016) due to their generality. Compared with direct amortization in Eq. 6.9, we use iterative amortized optimizers of the general form λ ← 𝑓𝜙 (s, λ, ∇λ J ), (6.10) also shown in Figure 6.1 (Right), where 𝑓𝜙 is a deep network and λ are the policy distribution parameters. For example, if 𝜋 = N (a; µ, diag(σ 2 )), then λ ≡ [µ, σ]. Technically, s is redundant, as the state dependence is already captured in J , but this can empirically improve performance (Chapter 3). In practice, the update is carried out using a “highway” gating operation (Hochreiter and Schmidhuber, 1997; Srivastava, Greff, and Schmidhuber, 2015). Denoting ω𝜙 ∈ [0, 1] as the gate and δ𝜙 as the update, both of which are output by 𝑓𝜙 , the gating operation is expressed as λ ← ω𝜙 λ + (1 − ω𝜙 ) δ𝜙 , (6.11) where denotes elementwise multiplication. This update is typically run for a fixed number of steps, and, as with a direct policy, the iterative optimizer is trained using stochastic gradient estimates of ∇𝜙 J , obtained through the pathwise derivative estimator (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014; Heess et al., 2015). Because the gradients ∇λ J must be estimated online, i.e., during policy optimization, this scheme requires some way of estimating J online, e.g., through a parameterized 𝑄-value network (V. Mnih, Kavukcuoglu, et al., 2013; Lillicrap et al., 2016) or a differentiable model (Heess et al., 2015). 99 Benefits of Iterative Amortization Reduced Amortization Gap. Iterative amortized optimizers are more flexible than their direct counterparts, incorporating feedback from the objective during policy optimization (Algorithm 4), rather than only after optimization (Algorithm 3). Increased flexibility improves the accuracy of optimization, thereby tightening the variational bound (Chapters 3 & 4). We see this flexibility in Figure 6.1 (Left), where an iterative amortized policy network iteratively refines the policy estimate (•), quickly arriving near the optimal estimate. Multiple Estimates. Iterative amortization, by using stochastic gradients and random initialization, can traverse the optimization landscape. As with any iterative optimization scheme, this allows iterative amortization to obtain multiple valid estimates, referred to as “multi-stability” in the perception literature (Greff et al., 2019). We illustrate this capability across two action dimensions in Figure 6.2 for a state in the Ant-v2 MuJoCo environment. Over multiple policy optimization runs, iterative amortization finds multiple modes, sampling from two high-value regions of the action space. This provides increased flexibility in action exploration, despite only using a uni-modal policy distribution. Generalization Across Objectives. Iterative amortization uses the gradients of the objective during optimization, i.e., feedback, allowing it to potentially generalize to new or updated objectives. We see this in Figure 6.1 (Left), where iterative amortization, despite being trained with a different value estimator, is capable of generalizing to this new objective. We demonstrate this capability further in Section 6.4. This opens the possibility of accurately and efficiently performing policy optimization in new settings, e.g., a rapidly changing model or new tasks. Consideration: Mitigating Value Overestimation Why are more powerful policy optimizers typically not used in practice? As we now describe, part of the issue stems from value overestimation. Model-free approaches generally estimate 𝑄 𝜋 using function approximation and temporal difference learning. However, this has the pitfall of value overestimation, i.e., positive bias in the estimate, b𝜋 (Thrun and Schwartz, 1993). This issue is tied to uncertainty in the value estimate, 𝑄 though it is distinct from optimism under uncertainty. If the policy can exploit regions of high uncertainty, the resulting target values will introduce positive bias into the estimate. More flexible policy optimizers exacerbate the problem, exploiting 100 1.0 12.5 410 10.0 πφ(a|s, O) 3 Opt. Run 1 2 3 4 7.5 0.0 J tanh(µ6) 0.5 405 −0.5 −1.0 −1.0 400 −0.5 0.0 tanh(µ2) 0.5 1.0 395 2 5.0 1 2.5 0.0 0 −1 0 Act. Dim. 2 1 −1 0 1 Act. Dim. 6 Figure 6.2: Estimating Multiple Policy Modes. Unlike direct amortization, which is restricted to a single estimate, the stochasticity of iterative optimization allows iterative amortization to effectively sample from multiple high-value modes in the action space. This capability is shown for a particular state in Ant-v2, showing multiple optimization runs across two action dimensions (Left). Each colored square denotes an initialization. The optimizer finds both modes, with the assigned density plotted on the Right. This capability provides increased flexibility in action exploration. this uncertainty to a greater degree. Further, a rapidly changing policy increases the difficulty of value estimation (Rajeswaran, Mordatch, and V. Kumar, 2020). Various techniques have been proposed for mitigating value overestimation in deep RL. The most prominent technique, double deep 𝑄-network (Van Hasselt, Guez, and Silver, 2016) maintains two 𝑄-value estimates (Van Hasselt, 2010), attempting to decouple policy optimization from value estimation. Fujimoto, Hoof, and Meger, 2018 apply and improve upon this technique for actor-critic settings, estimating the target 𝑄-value as the minimum of two 𝑄-networks, 𝑄 𝜓1 and 𝑄 𝜓2 : b𝜋 (s, a) = min 𝑄 𝜓 0 (s, a), 𝑄 𝑖 𝑖=1,2 where 𝜓𝑖0 denotes the “target” network parameters. As noted by Fujimoto, Hoof, and Meger, 2018, this not only counteracts value overestimation, but also penalizes high-variance value estimates, because the minimum decreases with the variance of the estimate. Ciosek et al., 2019 noted that, for a bootstrapped ensemble of two 𝑄-networks, the minimum operation can be interpreted as estimating b𝜋 (s, a) = 𝜇𝑄 (s, a) − 𝛽𝜎𝑄 (s, a), 𝑄 101 200 100 DKL(πnew||πold) b π − Qπ Q 150 103 direct, β = 1 iterative, β = 1 iterative, β = 2.5 102 50 101 0 −50 0.0 0.5 1.0 1.5 Million Steps (a) 2.0 2.5 3.0 100 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Million Steps (b) Figure 6.3: Mitigating Value Overestimation. Using the same value estimation setup (𝛽 = 1), shown on Ant-v2, iterative policy optimization results in (a) higher value overestimation bias and (b) a more rapidly changing policy as compared with direct policy optimization. Increasing 𝛽 helps to mitigate these issues by further penalizing variance in the value estimate. with mean and standard deviation 1Õ 𝜇𝑄 (s, a) ≡ 𝑄 𝜓 0 (s, a), 2 𝑖=1,2 𝑖 " #  2 1/2 1Õ 𝜎𝑄 (s, a) ≡ , 𝑄 𝜓𝑖0 (s, a) − 𝜇𝑄 (s, a) 2 𝑖=1,2 and 𝛽 = 1. Thus, to further penalize high-variance value estimates, preventing value overestimation, we can increase 𝛽. For large 𝛽, however, value estimates become overly pessimistic, negatively impacting training. Thus, 𝛽 reduces target value variance at the cost of increased bias. Due to the flexibility of iterative amortization, the default 𝛽 = 1 results in increased value bias (Figure 6.3a) and a more rapidly changing policy (Figure 6.3b) as compared with direct amortization. Further penalizing high-variance target values with 𝛽 = 2.5 reduces value overestimation and improves policy stability. For details, see Marino, Piché, et al., 2020. Recent techniques for mitigating overestimation have been proposed, such as adjusting the temperature, 𝛼 (Fox, 2019). In offline RL, this issue has been tackled through the action prior (Scott Fujimoto, David Meger, and Precup, 2019; A. Kumar, J. Fu, et al., 2019; Wu, Tucker, and Nachum, 2019) or by altering 𝑄-network training (Agarwal, Schuurmans, and Norouzi, 2019; A. Kumar, Zhou, et al., 2020). While such techniques could be used here, increasing 𝛽 provides a simple solution with no additional computational overhead. Opt. Iter. 1 2 3 4 5 0.5 0.0 −0.5 0 5 10 15 20 Environment Time Step (a) Policy 25 Opt. Iter. 1 2 3 4 5 40 ∆J Action Dim. 5 102 20 0 0 5 10 15 20 25 Environment Time Step (b) Improvement Figure 6.4: Policy Optimization. Visualization over time steps of (a) one dimension of the policy distribution and (b) the improvement in the objective, Δ J , across policy optimization iterations. 6.4 Experiments Setup To focus specifically on policy optimization, we implement iterative amortized policy optimization using the setup of soft actor-critic (SAC) described by Haarnoja, Zhou, Hartikainen, et al., 2018. This involves using two 𝑄-networks, uniform action prior, 𝑝 𝜃 (a|s) = U (−1, 1), and an automatic tuning scheme for the temperature, 𝛼. In our experiments, “direct” refers to direct amortized policy optimization employed in SAC, i.e., a direct policy network, and “iterative” refers to iterative amortized policy optimization. Both approaches use the same network architecture, adjusting only the number of inputs and outputs to accommodate gradients, current policy estimates, and gated updates (Sec. 6.3). Unless otherwise stated, we use 5 iterations per time step for iterative amortization, as in Chapter 3. For additional details, we refer to Marino, Piché, et al., 2020 and Haarnoja, Zhou, Abbeel, et al., 2018; Haarnoja, Zhou, Hartikainen, et al., 2018. Accompanying code is available at github.com/joelouismarino/variational_rl. Analysis Visualizing Policy Optimization In Figure 6.1 (Left), we visualize the trajectory of iterative amortized policy optimization along two dimensions of the policy mean on a state from Hopper-v2. Through iterative optimization, the network arrives near the optimum. Notably, this is performed with a value function trained using a different, direct policy, demonstrating generalization to other optimization landscapes. In Figure 6.4, we visualize iterative refinement using a single action dimension from Ant-v2 across time steps. The refinements in Figure 6.4a give rise to the objective improvements in Figure 6.4b. 0 0 1 2 15000 10000 5000 0 3 0 Million Steps 1 2 3 Cumulative Reward 2000 Cumulative Reward 4000 Cumulative Reward Cumulative Reward 103 6000 4000 2000 0 0 Million Steps (a) Hopper-v2 1 2 8000 direct (SAC) 4000 direct + NF iterative 0 3 0 Million Steps (b) HalfCheetah-v2 1 2 3 Million Steps (c) Walker2d-v2 (d) Ant-v2 Figure 6.5: Performance Comparison. Iterative amortized policy optimization performs comparably with or better than direct amortized policies across a range of MuJoCo environments. Performance curves show the mean and ± standard deviation over 5 random seeds. 10−2 0 1 2 3 10−1 direct (SAC) direct + NF 10−2 0 Million Steps (a) Hopper-v2 1 2 3 Million Steps (b) HalfCheetah-v2 100 Amortization Gap 10−1 100 Amortization Gap 100 Amortization Gap Amortization Gap 100 10−1 10−2 0 1 2 Million Steps (c) Walker2d-v2 3 10−1 iterative, 5 it. iterative, 10 it. 10−2 0 1 2 3 Million Steps (d) Ant-v2 Figure 6.6: Amortization Gap. Estimated amortization gaps per step for direct and iterative amortized policy optimization. Iterative amortization achieves comparable or lower gaps across environments. Gaps are estimated using stochastic gradientbased optimization over 100 random states. Curves show the mean and ± standard deviation over 5 random seeds. Performance Comparison We evaluate iterative amortized policy optimization on the suite of MuJoCo (Todorov, Erez, and Tassa, 2012) continuous control tasks from OpenAI gym (Brockman et al., 2016). In Figures 6.5 & 6.7, we compare the cumulative reward, i.e., return, of direct and iterative amortized policy optimization across environments. Each curve shows the mean and ± standard deviation of 5 random seeds. In all cases, iterative amortized policy optimization matches or outperforms the baseline direct amortized method, both in sample efficiency and final performance. Across environments, iterative amortization also yields more consistent, i.e., lower variance, performance. This suggests that iterative amortization is able to flexibly explore the optimization landscape, consistently arriving at improved policy estimates. Multiple Policy Modes To better understand the exploration benefits of iterative amortization, we also compare with direct amortization with a multi-modal policy distribution, formed using inverse autoregressive flows (Kingma, Salimans, et al., 2016), a type of normalizing flow 1000 500 0 10 20 30 5000 0 0 10 Thousand Steps 100 50 1 2 Million Steps (d) Swimmer-v2 30 0 −50 −100 −150 −200 .0 (b) InvertedDoublePendulum-v2 Cumulative Reward Cumulative Reward (a) InvertedPendulum-v2 0 0 20 Thousand Steps 3 150000 100000 50000 0 direct iterative 1 2 Million Steps (e) HumanoidStandup-v2 3 0.2 0.4 Million Steps (c) Reacher-v2 Cumulative Reward 0 10000 Cumulative Reward Cumulative Reward Cumulative Reward 104 6000 4000 2000 0 0 2 4 6 8 10 Million Steps (f) Humanoid-v2 Figure 6.7: Additional Performance Comparison. Results are shown on the remaining MuJoCo environments from OpenAI gym. Performance curves show the mean and ± standard deviation over 5 random seeds. (NF). Using a multi-modal policy reduces the performance deficiencies of direct amortization on Hopper-v2 and Walker2d-v2, indicating that much of the benefit of iterative amortization is due to lifting direct amortization’s restriction to a single, uni-modal policy estimate. Yet, direct + NF still struggles on HalfCheetah-v2 compared with iterative amortization (Fig. 6.5b), suggesting that more complex, multi-modal distributions are not the only consideration. To confirm that iterative amortization has captured multiple policy modes, at the end of training, we take Walker2d-v2 and histogram the distances between policy means across separate runs of policy optimization per state (Fig. 6.8a). For the state with the largest distance, we plot 2D projections of the optimization objective across action dimensions in Figure 6.8b, as well as the policy density across 10 optimization runs (Fig. 6.8c). We see that a subset of states indeed still retain multiple policy modes. Amortization Gap To evaluate policy optimization accuracy, we estimate per-step amortization gaps, performing additional iterations of gradient ascent on J w.r.t. the policy parameters, λ ≡ [µ, σ] (see Appendix A.3). To analyze generalization, we also evaluate the iterative agents trained with 5 iterations for an additional 5 amortized iterations. Results are shown in Figure 6.6. We emphasize that it is challenging to directly 105 Dim. 2 Dim. 3 Dim. 4 Dim. 5 Dim. 6 Dim. 1 Dim. 2 100 Dim. 3 10−1 Dim. 4 10−2 0.0 0.5 1.0 1.5 Dim. 5 || tanh (µ(i))− tanh (µ(j))||2 (a) πφ(a|s, O) 20 10 20 2 5 10 0 −1 (b) 10 0 1 Dim. 1 5 10 0 −1 0 1 Dim. 2 0 −1 0 0 −1 1 Dim. 3 0 1 Dim. 4 0 −1 5 0 1 Dim. 5 0 −1 0 1 Dim. 6 (c) 0 0 1 2 Million Steps (a) Hopper-v2 3 15000 10000 5000 0 0 1 2 3 Million Steps (b) HalfCheetah-v2 Cumulative Reward 2000 Cumulative Reward 4000 Cumulative Reward Cumulative Reward Figure 6.8: Multiple Policy Modes. (a) Histogram of distances between policy means (µ) across optimization runs (𝑖 and 𝑗) over seeds and states on Walker2d-v2 at 3 million environment steps. For the state with the largest distance, (b) shows the projected optimization surface on each pair of action dimensions, and (c) shows the policy density for 10 optimization runs. 6000 4000 2000 0 0 1 2 Million Steps (c) Walker2d-v2 3 8000 1 iter. 2 iter. 5 iter. 4000 0 1 2 3 Million Steps (d) Ant-v2 Figure 6.9: Varying Iterations During Training. Performance of iterative amortized policy optimization for varying numbers of iterations during training. Increasing the number of iterations generally results in improvements. Curves show the mean and ± standard deviation over 5 random seeds. compare amortization gaps across optimization schemes, as these involve different value functions, and therefore different objectives. Likewise, we estimate the amortization gap using the learned 𝑄-networks, which may be biased (Figure 6.3). Nevertheless, we find that iterative amortized policy optimization achieves, on average, lower amortization gaps than direct amortization across all environments. Additional amortized iterations at evaluation yield further estimated improvement, demonstrating generalization beyond the optimization horizon used during training. The amortization gaps are small relative to the objective, playing a negligible role in evaluation performance. Rather, improved policy optimization is helpful for training, allowing the agent to explore states where value estimates are highest. To 106 10−2 0 1 2 3 10 1 iter. 2 iter. 5 iter. 10−1 10−2 Million Steps 0 1 2 3 0 10−1 10−2 Million Steps (a) Hopper-v2 10 Amortization Gap 10−1 0 Amortization Gap 10 Amortization Gap Amortization Gap 10 0 0 1 2 3 0 10−1 10−2 Million Steps (b) HalfCheetah-v2 0 1 2 3 Million Steps (c) Walker2d-v2 (d) Ant-v2 Figure 6.10: Amortization Gap of Varying Iterations During Training. Corresponding amortization gaps for varying numbers of iterations during training. We generally see that increasing the number of iterations generally reduces the amortization gap. 270 100 200 Opt. Iteration (a) Hopper-v2 442 565 J 480 J J Adam CEM It. Amort. J 272 0 443 500 274 560 441 440 555 460 0 100 200 Opt. Iteration (b) HalfCheetah-v2 0 100 200 Opt. Iteration (c) Walker2d-v2 0 100 200 Opt. Iteration (d) Ant-v2 Figure 6.11: Comparison with Iterative Optimizers. Average estimated objective over policy optimization iterations, comparing with Adam (Kingma and Ba, 2014) and CEM (Rubinstein and Kroese, 2013). These iterative optimizers require over an order of magnitude more iterations to reach comparable performance with iterative amortization, making them impractical in many applications. probe this further, we train iterative amortized policy optimization while varying the number of iterations per step in {1, 2, 5}, yielding optimizers with varying degrees of accuracy. Note that each optimizer is, in theory, capable of finding multiple modes. In Figure 6.9, we see that training with additional iterations improves performance and optimization accuracy. Walker2d-v2 provides an interesting example. Even with a single iteration, we see that iterative amortization outperforms direct amortization, suggesting that multi-modality is the dominant factor for improved performance here. Yet, 1 iteration is slightly worse compared with 2 and 5 iterations early in training, both in terms of performance and optimization. As the amortization gap decreases later in training, we see that the performance gap ultimately decreases. Further work could help to analyze this process in even more detail. We stress that the exact form of this relationship depends on the 𝑄-value estimator and other factors. Comparison with Iterative Optimizers Iterative amortized policy optimization obtains the accuracy benefits of iterative optimization while retaining the efficiency benefits of amortization. We now compare iterative amortization with two popular iterative optimizers: Adam (Kingma and Ba, 107 2014), a gradient-based optimizer, and cross-entropy method (CEM) (Rubinstein and Kroese, 2013), a gradient-free optimizer. We collect 100 states for each seed in each environment from the model-free experiments. For each optimizer, we optimize the variational objective, J , starting from the same initialization. Tuning the step size, we found that 0.01 yielded the steepest improvement without diverging for both Adam and CEM. Gradients are evaluated with 10 action samples. For CEM, we sample 100 actions and fit a Gaussian mean and variance to the top 10 samples. This is comparable with QT-Opt (Kalashnikov et al., 2018), which draws 64 samples and retains the top 6 samples. The results, averaged across states and random seeds, are shown in Figure 6.11. CEM (gradient-free) is less efficient than Adam (gradient-based), which is unsurprising, especially considering that Adam effectively approximates higher-order curvature through momentum terms. However, Adam and CEM both require over an order of magnitude more iterations to reach comparable performance with iterative amortization. While iterative amortized policy optimization does not always obtain asymptotically optimal estimates, we note that these networks were trained with only 5 iterations, yet continue to improve and remain stable far beyond this limit. Finally, comparing wall clock time for each optimizer, iterative amortization is only roughly 1.25× slower than CEM and 1.15× slower than Adam, making iterative amortization still substantially more efficient. Generalizing to Model-Based Values Direct amortization is a purely feedforward process and is therefore incapable of generalizing to new objectives. In contrast, because iterative amortization is formulated through gradient-based feedback, such optimizers may be capable of generalizing to new objective estimators. To demonstrate this capability further, we apply iterative amortization with model-based value estimators, using a learned deterministic model on HalfCheetah-v2 (see Appendix A.5). In Figure 6.12a, we see that iterative amortization slightly outperforms direct amortization in this setting. The resulting policy optimization procedure refines planned trajectories, shown for a single state dimension in Figure 6.12b, yielding corresponding improvements (Fig. 6.12c). We evaluate the generalizing capabilities in Figure 6.12d by transferring the policy optimizer from a model-free agent to a model-based agent. Iterative amortization generalizes to these new value estimates, instantly recovering the performance of the model-based agent. This highlights the opportunity for instantly 5000 0 0.00 direct iterative 0.25 0.50 0.75 Million Steps (a) 1.00 0.0 −0.1 0 1 2 3 4 Model Rollout Step (b) 5 Cumulative Reward 10000 Opt. Iter. 1 2 3 4 5 580 J 0.1 15000 State Dim. 2 Cumulative Reward 108 570 560 0 1 2 3 4 5 15000 10000 MF MB MF → MB 5000 0 0.00 0.25 0.50 0.75 Opt. Iteration Million Steps (c) (d) 1.00 Figure 6.12: Optimizing Model-Based Value Estimates. (a) Performance comparison of direct and iterative amortization using model-based value estimates. (b) Planned trajectories over policy optimization iterations. (c) The corresponding estimated objective increases over iterations. (d) Zero-shot transfer of iterative amortization from model-free (MF) to model-based (MB) estimates. incorporating new tasks, goals, or model estimates into policy optimization. 6.5 Discussion In this chapter, we applied the technique of learned negative feedback to control, arriving at iterative amortized policy optimization, a flexible and powerful policy optimization technique. In so doing, we have highlighted several limitations of direct amortization: 1) limited accuracy, as quantified by the amortization gap, 2) restriction to a single estimate, limiting exploration, and 3) inability to generalize to new objectives, limiting the transfer of these policy optimizers. As confirmed through our empirical analysis on benchmark continuous control environments, iterative amortization provides a step toward improving each of these restrictions, with accompanying improvements in performance over current direct amortization methods. Thus, iterative amortization provides a drop-in replacement and improvement over direct policy networks in deep reinforcement learning. Although we have discussed three separate limitations of direct amortization, these factors are highly interconnected. By broadening policy optimization to an iterative procedure, we automatically obtain a potentially more accurate and general policy optimizer, with the capability of obtaining multiple modes. While our analysis suggests that the improved exploration resulting from multiple modes is the primary factor affecting performance, future work could tease out these effects further and assess the relative contributions of these improvements in additional environments. We are hopeful that iterative amortized policy optimization, by providing a more powerful, exploratory, and general optimizer, will enable a range of improved reinforcement learning algorithms. 109 References Abdolmaleki, Abbas et al. (2018). “Maximum a Posteriori Policy Optimisation”. In: International Conference on Learning Representations. Agarwal, Rishabh, Dale Schuurmans, and Mohammad Norouzi (2019). “An Optimistic Perspective on Offline Reinforcement Learning”. In: arXiv preprint arXiv:1907.04543. Ahmed, Zafarali et al. (2019). “Understanding the impact of entropy on policy optimization”. In: International Conference on Machine Learning, pp. 151–160. Amos, Brandon, Samuel Stanton, et al. (2020). “On the model-based stochastic value gradient for continuous reinforcement learning”. In: arXiv preprint arXiv:2008.12775. Amos, Brandon and Denis Yarats (2020). “The differentiable cross-entropy method”. In: International Conference on Machine Learning. Andrychowicz, Marcin et al. (2016). “Learning to learn by gradient descent by gradient descent”. In: Advances in Neural Information Processing Systems (NIPS), pp. 3981–3989. Astrom, Karl Johan and Richard M Murray (2008). Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press. Attias, Hagai (2003). “Planning by probabilistic inference.” In: AISTATS. Citeseer. Bharadhwaj, Homanga, Kevin Xie, and Florian Shkurti (2020). “Model-Predictive Planning via Cross-Entropy and Gradient-Based Optimization”. In: Learning for Dynamics and Control, pp. 277–286. Botvinick, Matthew and Marc Toussaint (2012). “Planning as inference”. In: Trends in cognitive sciences 16.10, pp. 485–488. Brockman, Greg et al. (2016). “Openai gym”. In: arXiv preprint arXiv:1606.01540. Byravan, Arunkumar et al. (2020). “Imagined Value Gradients: Model-Based Policy Optimization with Tranferable Latent Dynamics Models”. In: Conference on Robot Learning, pp. 566–589. Chua, Kurtland et al. (2018). “Deep reinforcement learning in a handful of trials using probabilistic dynamics models”. In: Advances in Neural Information Processing Systems, pp. 4754–4765. Ciosek, Kamil et al. (2019). “Better exploration with optimistic actor critic”. In: Advances in Neural Information Processing Systems, pp. 1787–1798. Clavera, Ignasi, Yao Fu, and Pieter Abbeel (2020). “Model-Augmented ActorCritic: Backpropagating through Paths”. In: International Conference on Learning Representations. Cooper, Gregory F (1988). “A method for using belief networks as influence diagrams”. In: Fourth Workshop on Uncertainty in Artificial Intelligence. 110 Cremer, Chris, Xuechen Li, and David Duvenaud (2018). “Inference Suboptimality in Variational Autoencoders”. In: International Conference on Machine Learning, pp. 1078–1086. Dayan, Peter and Geoffrey E Hinton (1997). “Using expectation-maximization for reinforcement learning”. In: Neural Computation 9.2, pp. 271–278. Fox, Roy (2019). “Toward Provably Unbiased Temporal-Difference Value Estimation”. In: Optimization Foundations for Reinforcement Learning Workshop at NeurIPS. Fox, Roy, Ari Pakman, and Naftali Tishby (2016). “Taming the noise in reinforcement learning via soft updates”. In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence. AUAI Press, pp. 202–211. Fujimoto, S, H van Hoof, and D Meger (2018). “Addressing function approximation error in actor-critic methods”. In: Proceedings of Machine Learning Research 80, pp. 1587–1596. Fujimoto, Scott, David Meger, and Doina Precup (2019). “Off-policy deep reinforcement learning without exploration”. In: International Conference on Machine Learning, pp. 2052–2062. Gershman, Samuel and Noah Goodman (2014). “Amortized inference in probabilistic reasoning”. In: Proceedings of the Cognitive Science Society. Vol. 36. 36. Greff, Klaus et al. (2019). “Multi-Object Representation Learning with Iterative Variational Inference”. In: International Conference on Machine Learning, pp. 2424– 2433. Guez, Arthur et al. (2019). “An Investigation of Model-Free Planning”. In: International Conference on Machine Learning, pp. 2464–2473. Haarnoja, Tuomas, Kristian Hartikainen, et al. (2018). “Latent Space Policies for Hierarchical Reinforcement Learning”. In: International Conference on Machine Learning, pp. 1846–1855. Haarnoja, Tuomas, Haoran Tang, et al. (2017). “Reinforcement learning with deep energy-based policies”. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, pp. 1352–1361. Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, et al. (2018). “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”. In: International Conference on Machine Learning, pp. 1856–1865. Haarnoja, Tuomas, Aurick Zhou, Kristian Hartikainen, et al. (2018). “Soft actor-critic algorithms and applications”. In: arXiv preprint arXiv:1812.05905. Hafner, Danijar et al. (2019). “Learning Latent Dynamics for Planning from Pixels”. In: International Conference on Machine Learning, pp. 2555–2565. Heess, Nicolas et al. (2015). “Learning continuous control policies by stochastic value gradients”. In: Advances in Neural Information Processing Systems, pp. 2944– 2952. 111 Henaff, Mikael, William F Whitney, and Yann LeCun (2017). “Model-based planning with discrete and continuous actions”. In: arXiv preprint arXiv:1705.07177. Hochreiter, Sepp and Jürgen Schmidhuber (1997). “Long short-term memory”. In: Neural computation 9.8, pp. 1735–1780. Kalashnikov, Dmitry et al. (2018). “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation”. In: arXiv preprint arXiv:1806.10293. Kalman, Rudolph Emil (1960). “A new approach to linear filtering and prediction problems”. In: Journal of Basic Engineering 82.1, pp. 35–45. Kim, Yoon et al. (2018). “Semi-Amortized Variational Autoencoders”. In: Proceedings of the International Conference on Machine Learning (ICML). Kingma, Durk P and Jimmy Ba (2014). “Adam: A method for stochastic optimization”. In: arXiv preprint arXiv:1412.6980. Kingma, Durk P, Tim Salimans, et al. (2016). “Improved variational inference with inverse autoregressive flow”. In: Advances in neural information processing systems, pp. 4743–4751. Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Kumar, Aviral, Justin Fu, et al. (2019). “Stabilizing off-policy q-learning via bootstrapping error reduction”. In: Advances in Neural Information Processing Systems, pp. 11784–11794. Kumar, Aviral, Aurick Zhou, et al. (2020). “Conservative Q-Learning for Offline Reinforcement Learning”. In: arXiv preprint arXiv:2006.04779. Lee, Keuntaek, Kamil Saigol, and Evangelos A Theodorou (2019). “Safe end-to-end imitation learning for model predictive control”. In: International Conference on Robotics and Automation. Levine, Sergey (2018). “Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review”. In: arXiv preprint arXiv:1805.00909. Lillicrap, Timothy P et al. (2016). “Continuous control with deep reinforcement learning”. In: International Conference on Learning Representations. Marino, Joseph, Alexandre Piché, Alessandro Davide Ialongo, and Yisong Yue (2020). “Iterative Amortized Policy Optimization”. In: Preprint. url: https: //arxiv.org/abs/2010.10670. Marino, Joseph, Yisong Yue, and Stephan Mandt (2018). “Iterative Amortized Inference”. In: International Conference on Machine Learning, pp. 3403–3412. url: http://proceedings.mlr.press/v80/marino18a.html. Mnih, Andriy and Karol Gregor (2014). “Neural Variational Inference and Learning in Belief Networks”. In: International Conference on Machine Learning, pp. 1791– 1799. 112 Mnih, Volodymyr, Adria Puigdomenech Badia, et al. (2016). “Asynchronous methods for deep reinforcement learning”. In: International conference on machine learning, pp. 1928–1937. Mnih, Volodymyr, Koray Kavukcuoglu, et al. (2013). “Playing Atari With Deep Reinforcement Learning”. In: NIPS Deep Learning Workshop. Nagabandi, Anusha et al. (2018). “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning”. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 7559–7566. Piché, Alexandre et al. (2019). “Probabilistic Planning with Sequential Monte Carlo methods”. In: International Conference on Learning Representations. url: https://openreview.net/forum?id=ByetGn0cYX. Rajeswaran, Aravind, Igor Mordatch, and Vikash Kumar (2020). “A Game Theoretic Framework for Model Based Reinforcement Learning”. In: arXiv preprint arXiv:2004.07804. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Rivière, Benjamin et al. (2020). “GLAS: Global-to-Local Safe Autonomy Synthesis for Multi-Robot Motion Planning with End-to-End Learning”. In: IEEE Robotics and Automation Letters 5.3, pp. 4249–4256. Rubinstein, Reuven Y and Dirk P Kroese (2013). The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media. Schulman, John, Sergey Levine, et al. (2015). “Trust region policy optimization”. In: International Conference on Machine Learning, pp. 1889–1897. Schulman, John, Filip Wolski, et al. (2017). “Proximal policy optimization algorithms”. In: arXiv preprint arXiv:1707.06347. Silver, David et al. (2014). “Deterministic Policy Gradient Algorithms”. In: International Conference on Machine Learning, pp. 387–395. Srinivas, Aravind et al. (2018). “Universal Planning Networks: Learning Generalizable Representations for Visuomotor Control”. In: International Conference on Machine Learning, pp. 4732–4741. Srivastava, Rupesh K, Klaus Greff, and Jürgen Schmidhuber (2015). “Training very deep networks”. In: Advances in neural information processing systems (NIPS), pp. 2377–2385. Sutton, Richard S and Andrew G Barto (2018). Reinforcement learning: An introduction. MIT press. 113 Tang, Yunhao and Shipra Agrawal (2018). “Boosting Trust Region Policy Optimization by Normalizing Flows Policy”. In: arXiv preprint arXiv:1809.10326. Thrun, Sebastian and Anton Schwartz (1993). “Issues in using function approximation for reinforcement learning”. In: Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum. Tirumala, Dhruva et al. (2019). “Exploiting Hierarchy for Learning and Transfer in KL-regularized RL”. In: arXiv preprint arXiv:1903.07438. Todorov, Emanuel (2008). “General duality between optimal control and estimation”. In: Decision and Control, 2008. CDC 2008. 47th IEEE Conference on. IEEE, pp. 4286–4292. Todorov, Emanuel, Tom Erez, and Yuval Tassa (2012). “Mujoco: A physics engine for model-based control”. In: Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, pp. 5026–5033. Toussaint, Marc and Amos Storkey (2006). “Probabilistic inference for solving discrete and continuous state Markov Decision Processes”. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp. 945–952. Van Hasselt, Hado (2010). “Double Q-learning”. In: Advances in neural information processing systems, pp. 2613–2621. Van Hasselt, Hado, Arthur Guez, and David Silver (2016). “Deep reinforcement learning with double q-learning”. In: Thirtieth AAAI conference on artificial intelligence. Verma, Deepak and Rajesh PN Rao (2006). “Goal-based imitation as probabilistic inference over graphical models”. In: Advances in neural information processing systems. Citeseer, pp. 1393–1400. Wiener, Norbert (1948). Cybernetics or Control and Communication in the Animal and the Machine. MIT press. Williams, Ronald J (1992). “Simple statistical gradient-following algorithms for connectionist reinforcement learning”. In: Reinforcement Learning. Springer, pp. 5–32. Wu, Yifan, George Tucker, and Ofir Nachum (2019). “Behavior regularized offline reinforcement learning”. In: arXiv preprint arXiv:1911.11361. 114 Chapter 7 SEQUENTIAL AUTOREGRESSIVE FLOW-BASED POLICIES learned feedforward control 31 Guerra, Alex and Joseph Marino (2020). “Sequential Autoregressive Flow-Based Policies”. In: ICML workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models. 7.1 Introduction In this chapter, we complete the table outlined in Chapter 1, applying learned feedforward processing to control. As opposed to purely feedback policies (Chapter 6), which must optimize the control objective from scratch at each time step, we consider policies with a feedforward component, predicting useful actions at future time steps. This is loosely inspired by the hierarchical organization of animal motor control (Merel, Botvinick, and Wayne, 2019), in which central pattern generator circuits (Marder and Bucher, 2001) provide low-level dynamical motor primitives, receiving transient top-down signals from higher-level motor areas (Shalit et al., 2012). This improves computational efficiency and speed, as feedback optimization (inference) is relied upon less and less as feedforward behavioral routines are learned. In formulating learned feedforward control policies, we again consider sequential autoregressive flows, as introduced in Chapter 5. Thus, the overall policy is composed of a dynamical feedforward component, conditioned on previous actions, and a feedback component, conditioned on the current state. We provide an initial set of 115 experiments exploring the use of these policies in continuous control environments, demonstrating improvements in performance as well as the ability to distill purely feedforward locomotion policies. 7.2 Autoregressive Flow-Based Policies Basic Setup We again consider the variational RL setup, as introduced in Chapter 6. As a brief review, we consider a Markov decision process (MDP), in which, at time 𝑡, an agent receives a state observation s𝑡 ∈ S and takes action a𝑡 ∈ A by sampling from a policy distribution, 𝜋. The agent then receives reward 𝑟 (s𝑡 , a𝑡 ) and the environment transitions to the next state s𝑡+1 ∼ 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ). By interpreting RL as variational inference (Ziebart, 2010; Levine, 2018), we arrive at a KL-regularized lower bound on the standard RL objective: " 𝑇 # Õ  𝜋(a |s , O) 𝑡 𝑡 J (𝜋) = E 𝑝env ,𝜋 𝛾 𝑡 𝑟 (s𝑡 , a𝑡 ) − 𝛼 log , (7.1) 𝑝(a |s ) 𝑡 𝑡 𝑡=1 where 𝛾 ∈ [0, 1) is the discount factor, 𝛼 is a Lagrange multiplier controlling the regularization weight, O is optimality (as introduced in the previous chapter), and 𝑝(a𝑡 |s𝑡 ) is an action prior. For simplicity, we again consider a uniform action prior in this chapter (see Section 7.4 for further discussion). Soft actor-critic (SAC) (Haarnoja, Zhou, Abbeel, et al., 2018) provides one instantiation of this setup, learning a 𝑄-network (critic) to estimate future terms in the objective, and a direct amortized policy network (actor) to optimize the objective. The critic is learned using temporal difference learning, with an ensemble of deep networks (Fujimoto, Hoof, and Meger, 2018), target value networks (Mnih et al., 2015), and an experience replay buffer (Lin, 1992). SAC estimates the policy using a deep network, denoted 𝜋 𝜙 , which typically takes the form of a conditional Gaussian: 𝜋 𝜙 (a𝑡 |s𝑡 , O) = N (a𝑡 ; µ𝜙 (s𝑡 ), diag(σ𝜙2 (s𝑡 ))). In bounded action spaces, tanh is typically applied to the policy samples (Haarnoja, Zhou, Abbeel, et al., 2018). The policy network parameters, 𝜙, are optimized by differentiating through the objective using the reparameterization gradient estimator (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). Motivation As noted in the previous chapter, performing policy optimization (inference) at the current time step only requires access to an estimate of the objective (Eq. 7.1). 116 a<t <latexit sha1_base64="iMo9bgRy40ENjhb5MoWCxV8H2sc=">AAAChnicdVHLTsJAFB3qC/EB6NLNRELiirQ+ggsXRDcuNQEhgaaZDlOYMNNpZm6NpOmXuNWP8m+cIgsQvclNTs59nZwbJoIbcN2vkrO1vbO7V96vHBweHVdr9ZMXo1JNWY8qofQgJIYJHrMecBBskGhGZChYP5w9FPX+K9OGq7gL84T5kkxiHnFKwFJBrTqSBKZhlJE8yO4gD2oNt+UuAm8CbwkaaBlPQb0UjMaKppLFQAUxZui5CfgZ0cCpYHlllBqWEDojEza0MCaSGT9bKM9x0zJjHCltMwa8YFcnMiKNmcvQdhY6ze9aQf5VG6YQ3foZj5MUWEx/DkWpwKBwYQMec80oiLkFhGputWI6JZpQsGatbep6flaIK9asnQ9lXmniVaaQkYB8W+8jYqLshan8h+Z0cyAxLLWuqrE10L7E+/2ATfBy2fKuWpfP143O/fI5ZXSGztEF8lAbddAjekI9RFGK3tEH+nTKTsu5cdo/rU5pOXOK1sLpfAOlW8h5</latexit> ( <latexit sha1_base64="mdeLchLY9vPvBC8/zTy2qbNpSww=">AAACfXicdVHLTgIxFC3jC/EFunTTSEhcmMkMmuCS4MYlJrwSmJBO6QwNfUzajpFM+Am3+mN+jXaABYjepMnJuY9zem+YMKqN530VnL39g8Oj4nHp5PTs/KJcuexpmSpMulgyqQYh0oRRQbqGGkYGiSKIh4z0w9lTnu+/EqWpFB0zT0jAUSxoRDEylhqMWjSO4Sgbl6ue6y0D7gJ/DapgHe1xpTAeTSROOREGM6T10PcSE2RIGYoZWZRGqSYJwjMUk6GFAnGig2xpeAFrlpnASCr7hIFLdrMjQ1zrOQ9tJUdmqn/ncvKv3DA10WOQUZGkhgi8EopSBo2E+e/hhCqCDZtbgLCi1ivEU6QQNnZHW5M6fpDl5vIxW/IhX5RqcJPJbSSGv23XIRZLqzDl/9AU7zYkmqR2q3JiF2hP4v8+wC7o1V3/3q2/PFSbrfVxiuAa3IBb4IMGaIJn0AZdgAED7+ADfBa+nZpz57irUqew7rkCW+E0fgAaGMWf</latexit> at <latexit sha1_base64="UEHjFduc+24G2yMIrW39dEMKk3k=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxqWBV6Awlk2ba0GQSkjtiGfobbvW3/BsztQvH6oULh3Nfh3NjLbgF3/+seNW19Y3N2lZ9e2d3b7/RPHi0KjOU9akSyjzHxDLBU9YHDoI9a8OIjAV7iqc3Rf3phRnLVfoAM80iScYpTzgl4KgwlAQmcZKT+RCGjZbf8ReBV0GwBC20jLthszIMR4pmkqVABbF2EPgaopwY4FSweT3MLNOETsmYDRxMiWQ2yhei57jtmBFOlHGZAl6wPydyIq2dydh1FiLt71pB/lUbZJBcRjlPdQYspd+HkkxgULhwAI+4YRTEzAFCDXdaMZ0QQyg4n0qbHoIoL8QVa0rnYzmvt/FPppChQb6W+4gYK3dhIv+hOV0d0JZlzlU1cga6lwS/H7AKHrud4KzTvT9v9a6Xz6mhI3SMTlCALlAP3aI71EcUafSG3tGHV/VOPd/rfrd6leXMISqFd/UFHhLG9g==</latexit> <latexit sha1_base64="PIcE15ob5qhe7HWVZQoV0KkQvHY=">AAACiXicdVFNT9tAEN2YUmigbQJHLisipJ4imyIV9RSVC0eQCCDFljVeT5IVu15rd1wRWfkrXNu/1H/DOskhIWWkkZ7efD29yUolHYXhv1aw82H3497+p/bB4ecvXzvdo3tnKitwKIwy9jEDh0oWOCRJCh9Li6AzhQ/Z01VTf/iN1klT3NGsxETDpJBjKYA8lXaO4kzXcY6KYJ7GNEWCtNML++Ei+DaIVqDHVnGTdltpnBtRaSxIKHBuFIUlJTVYkkLhvB1XDksQTzDBkYcFaHRJvRA/52eeyfnYWJ8F8QW7PlGDdm6mM9+pgabuba0h/1cbVTS+TGpZlBVhIZaHxpXiZHjjBM+lRUFq5gEIK71WLqZgQZD3a2PTXZTUjbhmzcb5TM/bZ3ydaWSUpJ83+0BNjL8w1e/QUmwPlA4r76rJvYH+JdHbB2yD+/N+9L1/fnvRG/xaPWefnbBT9o1F7AcbsGt2w4ZMsGf2wv6wv8FBEAWXwc9la9BazRyzjQiuXgGXDMnD</latexit> <latexit sha1_base64="1QlRqzgI9vtll2fiIvQldc+sPKU=">AAACiHicdVHLSgMxFE3Hd321unQTLAVXZUYFdSe6canQaqEzDJn0tg0mkyG5o5ahn+JWv8m/MVO7aK1eCDmc+zqcm2RSWPT9r4q3srq2vrG5Vd3e2d3br9UPHq3ODYcO11KbbsIsSJFCBwVK6GYGmEokPCXPt2X+6QWMFTpt4ziDSLFhKgaCM3RUXKuHiSrCBJBN4hBH7o9rDb/lT4Mug2AGGmQW93G9Eod9zXMFKXLJrO0FfoZRwQwKLmFSDXMLGePPbAg9B1OmwEbFVPuENh3TpwNt3EuRTtn5joIpa8cqcZWK4cj+zpXkX7lejoPLqBBpliOk/GfRIJcUNS2NoH1hgKMcO8C4EU4r5SNmGEdn18KkdhAVpbhyzML6RE2qTTrPlDIyVG+LdUwOtdswUv/Qgi83ZBZy56ruOwPdSYLfB1gGj6et4Kx1+nDeuL6ZHWeTHJFjckICckGuyR25Jx3CySt5Jx/k06t6vnfhXf2UepVZzyFZCO/mG4Z3yUs=</latexit> ✓ + ✓ ⇤ <latexit sha1_base64="Z+9aU0F2yHtHC/FjEFUaW/dEtUg=">AAACdnicdVHLSgMxFE3HV62vVpeCBEtREMpMFXRZdOOyhb6gHUomvW1Dk8mQZMQy9Avc6sf5Jy7NtF30oRcCh3Mf5+TeIOJMG9f9zjg7u3v7B9nD3NHxyelZvnDe0jJWFJpUcqk6AdHAWQhNwwyHTqSAiIBDO5i8pPn2GyjNZNgw0wh8QUYhGzJKjKXqd/180S2788DbwFuCIlpGrV/I9HsDSWMBoaGcaN313Mj4CVGGUQ6zXC/WEBE6ISPoWhgSAdpP5k5nuGSZAR5KZV9o8Jxd7UiI0HoqAlspiBnrzVxK/pXrxmb45CcsjGIDIV0IDWOOjcTpt/GAKaCGTy0gVDHrFdMxUYQau5y1SQ3PT1Jz6Zg1+UDMciW8yqQ2IiPe1+sIH0mrMBb/0IxuN0QaYrtVObALtCfxNg+wDVqVsndfrtQfitXn5XGy6BJdo1vkoUdURa+ohpqIIkAf6BN9ZX6cK6fk3CxKncyy5wKtheP+ApiLwrg=</latexit> <latexit sha1_base64="Nkk3T+niU2cCEWunvav3KuI5f5w=">AAACdnicdVHLSgMxFE3HV62vVpeCBEtRXJSZKuiy6MZlC31BO5RMetuGJpMhyYhl6Be41Y/zT1yaabvoQy8EDuc+zsm9QcSZNq77nXF2dvf2D7KHuaPjk9OzfOG8pWWsKDSp5FJ1AqKBsxCahhkOnUgBEQGHdjB5SfPtN1CaybBhphH4goxCNmSUGEvV7/r5olt254G3gbcERbSMWr+Q6fcGksYCQkM50brruZHxE6IMoxxmuV6sISJ0QkbQtTAkArSfzJ3OcMkyAzyUyr7Q4Dm72pEQofVUBLZSEDPWm7mU/CvXjc3wyU9YGMUGQroQGsYcG4nTb+MBU0ANn1pAqGLWK6Zjogg1djlrkxqen6Tm0jFr8oGY5Up4lUltREa8r9cRPpJWYSz+oRndbog0xHarcmAXaE/ibR5gG7QqZe++XKk/FKvPy+Nk0SW6RrfIQ4+oil5RDTURRYA+0Cf6yvw4V07JuVmUOpllzwVaC8f9BZZ5wrc=</latexit> ut <latexit sha1_base64="mVKEMNvj9Sfry61DI9HK+if7xTk=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxqWBV6Awlk2ba0GQSkjtiGfobbvW3/BsztQvH6oULh3Nfh3NjLbgF3/+seNW19Y3N2lZ9e2d3b7/RPHi0KjOU9akSyjzHxDLBU9YHDoI9a8OIjAV7iqc3Rf3phRnLVfoAM80iScYpTzgl4KgwlAQmcZJn8yEMGy2/4y8Cr4JgCVpoGXfDZmUYjhTNJEuBCmLtIPA1RDkxwKlg83qYWaYJnZIxGziYEslslC9Ez3HbMSOcKOMyBbxgf07kRFo7k7HrLETa37WC/Ks2yCC5jHKe6gxYSr8PJZnAoHDhAB5xwyiImQOEGu60YjohhlBwPpU2PQRRXogr1pTOx3Jeb+OfTCFDg3wt9xExVu7CRP5Dc7o6oC3LnKtq5Ax0Lwl+P2AVPHY7wVmne3/e6l0vn1NDR+gYnaAAXaAeukV3qI8o0ugNvaMPr+qder7X/W71KsuZQ1QK7+oLR7bHCg==</latexit> st <latexit sha1_base64="uvADgAvNfkx84xYZdD6H3tEw1HI=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxqWBV6Awlk2ba0GQSkjtiGfobbvW3/BsztQvH6oULh3Nfh3NjLbgF3/+seNW19Y3N2lZ9e2d3b7/RPHi0KjOU9akSyjzHxDLBU9YHDoI9a8OIjAV7iqc3Rf3phRnLVfoAM80iScYpTzgl4KgwlAQmcZLb+RCGjZbf8ReBV0GwBC20jLthszIMR4pmkqVABbF2EPgaopwY4FSweT3MLNOETsmYDRxMiWQ2yhei57jtmBFOlHGZAl6wPydyIq2dydh1FiLt71pB/lUbZJBcRjlPdQYspd+HkkxgULhwAI+4YRTEzAFCDXdaMZ0QQyg4n0qbHoIoL8QVa0rnYzmvt/FPppChQb6W+4gYK3dhIv+hOV0d0JZlzlU1cga6lwS/H7AKHrud4KzTvT9v9a6Xz6mhI3SMTlCALlAP3aI71EcUafSG3tGHV/VOPd/rfrd6leXMISqFd/UFQ4zHCA==</latexit> 40 Figure 7.1: Autoregressive Flow-Based Policy. An autoregressive flow-based policy converts samples from a base distribution, 𝜋 𝜙 (u𝑡 |s𝑡 ), using an affine transform with parameters β𝜃 (a<𝑡 ) and δ𝜃 (a<𝑡 ), into actions, a𝑡 . The affine transform (top) acts as a feedforward policy, purely conditioned on previous actions, whereas the base distribution (bottom) acts as a feedback policy. Each are parameterized by a separate deep network, with parameters 𝜃 and 𝜙, respectively. However, in fully-observable MDPs with continuous action spaces, it is common to use direct amortized policy networks, typically directly mapping the current state to a distribution over actions, i.e., 𝜋 𝜙 (a𝑡 |s𝑡 , O). This is because s𝑡 provides a compact description of the information necessary to perform policy optimization. Yet, this is ultimately a design choice, and we are free to condition direct amortization on any current or past variables: 𝜋 𝜙 (a𝑡 |s≤𝑡 , a<𝑡 , O). In this chapter, we specifically focus on conditioning on previous actions, parameterizing policies of the general form 𝜋 𝜙 (a𝑡 |s𝑡 , a<𝑡 , O). Although this may, at first, appear to be an odd design choice, there are several possible benefits to parameterizing policies with a feedforward component conditioned on previous actions. First, policy optimization can be computationally costly; having a feedforward policy component may reduce the computational burden on feedback optimization. Second, there may be communication constraints or delays from high-level state information; a feedforward policy may provide a reasonable control estimate quickly. Finally, learning to perform feedback optimization can be challenging; a feedforward policy provides a basis of dynamical motor primitives (Ijspeert, Nakanishi, and Schaal, 2002), which may simplify upstream control learning. 117 Formulation We now describe adapting sequential autoregressive flows (Chapter 5) to the control setting. To simplify notation, we omit the dependency on optimality, O, in the description. We consider a policy composed of a state-dependent base distribution, 𝜋 𝜙 (u𝑡 |s𝑡 ), over a latent variable, u, and a dynamical affine autoregressive (feedforward) component, defined by a shift, δ𝜃 (a<𝑡 ), and scale, β𝜃 (a<𝑡 ). To generate action a𝑡 , we sample the latent variable, u𝑡 ∼ 𝜋𝜃 (u𝑡 |s𝑡 ), then apply the affine transform, a𝑡 = β𝜃 (a<𝑡 ) u𝑡 + δ𝜃 (a<𝑡 ). (7.2) With the change of variables, the action probability is then −1 𝜋 𝜙,𝜃 (a𝑡 |s𝑡 , a<𝑡 ) = 𝜋 𝜙 (u𝑡 |s𝑡 ) Ö 𝛽𝜃,𝑖 (a<𝑡 ) , (7.3) 𝑖 where 𝛽𝜃,𝑖 is the 𝑖 th dimension of β𝜃 . As in SAC, we can also apply a final tanh transform if the action space is bounded in [−1, 1]. In the implementation presented here, the base distribution is a Gaussian, output by a deep network with parameters 𝜙, and, similarly, the affine transform parameters are output by a deep network with parameters 𝜃. The overall setup is shown in Figure 7.1. As we have noted multiple times throughout this thesis, if u𝑡 ∼ N (u𝑡 ; 0, I), then the affine flow is equivalent to an autoregressive Gaussian model, as Eq. 7.2 is the Gaussian reparameterization trick (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). For more general base distributions, this serves as a technique for adding temporal dependencies to output sequences, a1:𝑇 . Conversely, given a sequence, a1:𝑇 , the affine transform provides a mechanism for removing temporal dependencies, i.e., I (u1:𝑇 ) ≤ I (a1:𝑇 ), as seen in Chapter 5, thereby simplifying estimation in the space of u1:𝑇 . 7.3 Experiments We incorporate the proposed autoregressive feedforward component within soft actor-critic (SAC), which we refer to as autoregressive SAC (ARSAC). The affine transform parameters in ARSAC are conditioned on a varying number of previous actions, which we denote as, e.g., ARSAC-5 for 5 previous actions. Unless otherwise stated, we adopt the default hyperparameters from Haarnoja, Zhou, Abbeel, et al., 2018, as well as the automatic entropy-tuning scheme proposed in Haarnoja, Zhou, Hartikainen, et al., 2018. We evaluate ARSAC across a range of locomotion tasks in environments from the dm_control suite (Tassa et al., 2020). 1 2 3 0 0 1 Million Steps SAC ARSAC-5 500 250 0 0 1 2 3 Cumulative Reward 250 0 0 1 500 0 SAC ARSAC-5 0 1 2 3 750 500 SAC ARSAC-5 250 2 4 Million Steps (i) humanoid stand SAC ARSAC-5 250 0 0 1 500 SAC ARSAC-5 250 0 0 1 SAC ARSAC-5 500 250 2 4 Million Steps (j) humanoid walk 2 3 (d) walker run 2 3 1000 750 500 SAC ARSAC-5 250 0 0 1 2 3 Million Steps (g) quadruped walk 750 0 500 Million Steps 750 1000 0 750 Million Steps (f) cheetah run 1000 3 (c) walker walk 750 250 2 1000 Million Steps (e) swimmer swimmer6 0 500 1000 Million Steps 1000 Million Steps 0 3 (b) hopper stand Cumulative Reward Cumulative Reward (a) hopper hop 1000 750 2 SAC ARSAC-5 750 Million Steps Cumulative Reward 0 SAC ARSAC-5 250 Cumulative Reward 0 500 1000 Cumulative Reward 250 750 Cumulative Reward 500 1000 Cumulative Reward SAC ARSAC-5 750 Cumulative Reward 1000 Cumulative Reward Cumulative Reward 118 (h) quadruped run 1000 SAC ARSAC-5 750 500 250 0 0 2 4 Million Steps (k) humanoid run Figure 7.2: Performance Comparison, 2 × 256 (Default) Policy Network. Comparison between SAC and ARSAC with an action window of 5 time steps across dm_control suite environments. Each curve shows the mean and standard deviation across 5 random seeds. Performance Comparison In Figure 7.2, we compare the cumulative reward of SAC and ARSAC-5 throughout training, with 5 random seeds for each setup on each environment. On most environments, ARSAC-5 performs roughly as well or better than SAC, with improved sample efficiency and/or final performance. However, ARSAC-5 struggles on the more difficult tasks, humanoid walk and humanoid run. In Figure 7.3, we present performance results on these latter environments with varying autoregressive window sizes. We see that changing the size improves performance, however, this is still not able to bridge the performance gap on humanoid run. We hypothesize that the autoregressive transform will have a larger impact when the (state-dependent) base distribution is constrained in some way, either computationally or through delays. To probe this aspect, we decrease the size of the base distribution network, restricting the network to a single hidden layer with 32 units (the default size is 2 hidden layers, each with 256 units). The results, on a subset of environments, are shown in Figure 7.4, where we see that, indeed, the gap in performance slightly increases. 1000 Cumulative Reward Cumulative Reward 119 750 500 250 0 0 2 4 1000 SAC ARSAC-3 ARSAC-5 ARSAC-10 750 500 250 0 0 2 Million Steps 4 Million Steps (a) humanoid walk (b) humanoid run SAC ARSAC-5 500 250 0 0 1 2 3 1000 750 500 SAC ARSAC-5 250 0 0 Million Steps 750 500 0 SAC ARSAC-5 0 1 2 Million Steps (d) cheetah run 2 3 750 500 SAC ARSAC-5 250 0 0 3 750 500 0 SAC ARSAC-5 0 1 2 2 3 (c) walker run 1000 250 1 Million Steps (b) walker walk Cumulative Reward Cumulative Reward (a) hopper stand 1000 250 1 1000 Million Steps Cumulative Reward 750 Cumulative Reward 1000 Cumulative Reward Cumulative Reward Figure 7.3: Humanoid Performance Comparison, 2 × 256 (Default) Policy Network. Comparison between SAC and ARSAC with varying action window sizes on humanoid tasks from dm_control suite. Each curve shows the mean and standard deviation across 5 random seeds. 3 Million Steps (e) quadruped walk 1000 SAC ARSAC-5 750 500 250 0 0 1 2 3 Million Steps (f) quadruped run Figure 7.4: Performance Comparison, 1 × 32 Policy Network. Comparison between SAC and ARSAC with an action window of 5 time steps across dm_control suite environments. Each curve shows the mean and standard deviation across 5 random seeds. Policy Visualization & Feedforward Distillation We now analyze the policy distributions learned by ARSAC. On the left side of Figure 7.5, we plot the base distribution, 𝜋(u|s), as well as the autoregressive affine transform, δ𝜃 ± β𝜃 , and the actions (before applying the tanh transform). Arbitrarily, we only select the first action dimension for purposes of visualization. We see that the autoregressive flow contains relatively little temporal structure at the end of training. This is not entirely surprising, as there is nothing in the objective that explicitly requires the policy to be distilled into the autoregressive flow. Thus, while the flow is beneficial for training, it may not necessarily automatically provide dynamical “motor primitives,” part of the motivation for this approach. To explore this capability, we attempt to distill the policy entirely into the affine transform, arriving at a purely feedforward policy. We do so by restricting the base π(u|s) 5 0 −5 2.5 0.0 −2.5 tanh−1(a) δθ ± βθ 2.5 tanh−1(a) tanh−1(a) π(u|s) 120 0.0 −2.5 100 150 200 250 2.5 0.0 −2.5 100 150 Step π(u|s) tanh−1(a) π(u|s) tanh−1(a) 2.5 0.0 −2.5 tanh−1(a) δθ ± βθ 0.0 100 150 200 250 2.5 0.0 −2.5 100 150 250 (d) cheetah run (dist.), dim. 0 π(u|s) 5 0 −5 0.0 −2.5 tanh−1(a) δθ ± βθ 2.5 tanh−1(a) π(u|s) tanh−1(a) 200 Step (c) cheetah run, dim. 0 0.0 100 150 200 250 2.5 0.0 −2.5 tanh−1(a) δθ ± βθ 100 150 Step 200 250 Step (e) quadruped run, dim. 0 (f) quadruped run (dist.), dim. 0 5 π(u|s) 2.5 0 2 0 100 0.0 −2.5 tanh−1(a) δθ ± βθ tanh−1(a) π(u|s) tanh−1(a) δθ ± βθ Step tanh−1(a) 250 (b) walker run (dist.), dim. 0 5 0 −5 −2.5 200 Step (a) walker run, dim. 0 2.5 tanh−1(a) δθ ± βθ 150 200 Step (g) humanoid run, dim. 0 250 tanh−1(a) δθ ± βθ 2.5 0.0 −2.5 100 150 200 250 Step (h) humanoid run (dist.), dim. 0 Figure 7.5: Policy Visualization & Distillation. Left: Visualization of the base distribution and autoregressive flow for various action dimensions across various environments. Right: Visualization after policy distillation. distribution, progressively penalizing the L2 norm of the mean (µ𝜙 ) and log-standard deviation (log σ𝜙 ), thereby bringing the base distribution toward a standard Gaussian, N (u; 0, I). The results are shown on the right side of Figure 7.5, where we see that 121 the flow now contains nearly all of the temporal structure. In other words, these policies are effectively autoregressive Gaussian densities: 𝜋𝜃 (a𝑡 |a<𝑡 ) = N (a𝑡 ; δ𝜃 (a<𝑡 ), diag(β𝜃2 (a<𝑡 ))), (7.4) almost entirely independent of the state, s. Comparing the left and right columns, we see that the feedforward policies are more consistent overall, resulting from the limited temporal window of the affine flow. Thus, the autoregressive nature of the policy provides an inductive bias, resulting in policies with more regular rhythmic structure. As will be described in Chapter 8, this may hold some connections with central pattern generator circuits in biological neural systems, which produce rhythmic patterns in the absence of any state input (Marder and Bucher, 2001). 7.4 Discussion This chapter has presented a formulation of learned feedforward control based on sequential autoregressive flows. This approach decomposes the policy into a feedforward component, purely conditioned on previous actions, and a feedback component, conditioned on the current state, which are combined through an affine (linear) transform. In the initial set of experiment presented here on locomotion tasks, we have shown that such policies can outperform purely feedback-based counterparts. This is somewhat unsurprising, considering that these flow-based policies condition on additional variables and contain additional parameters. However, it is also somewhat surprising, considering that, in these environments, the current state is sufficient for estimating the optimal policy. This suggests that incorporating a feedforward policy component is a useful inductive bias for these tasks, which is reasonable, as they require periodic policies. Through qualitative analyses, we have observed that this periodic structure is not ultimately preserved in the flow. However, we have shown that it is feasible to distill the policy almost entirely into the flow. This may be advantageous when computational costs or temporal delays prohibit excessive amounts of feedback control. We also expect this approach to be more useful in environments with relatively minimal amounts of stochasticity (like dm_control), where state information can eventually be ignored. Future works may wish to consider other forms of normalizing flows, e.g., non-affine flows, flows with multiple transforms, or combining both spatial and temporal normalizing flows. We also recommend exploring the use of the prior, 𝑝(a𝑡 |·), as the feedforward component, as this will naturally preserve the dynamical structure of the policy. However, there are added challenges associated with training the prior, which will need to be resolved 122 (Abdolmaleki et al., 2018). At the very least, our results show that feedforward policies are practically possible, and exploring additional computational architectures for policies is a fruitful direction for future research. References Abdolmaleki, Abbas et al. (2018). “Relative entropy regularized policy iteration”. In: arXiv preprint arXiv:1812.02256. Fujimoto, S, H van Hoof, and D Meger (2018). “Addressing function approximation error in actor-critic methods”. In: Proceedings of Machine Learning Research 80, pp. 1587–1596. Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, et al. (2018). “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”. In: International Conference on Machine Learning, pp. 1856–1865. Haarnoja, Tuomas, Aurick Zhou, Kristian Hartikainen, et al. (2018). “Soft actor-critic algorithms and applications”. In: arXiv preprint arXiv:1812.05905. Ijspeert, Auke Jan, Jun Nakanishi, and Stefan Schaal (2002). “Movement imitation with nonlinear dynamical systems in humanoid robots”. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292). Vol. 2. IEEE, pp. 1398–1403. Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Levine, Sergey (2018). “Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review”. In: arXiv preprint arXiv:1805.00909. Lin, Long-Ji (1992). “Self-improving reactive agents based on reinforcement learning, planning and teaching”. In: Machine learning 8.3-4, pp. 293–321. Marder, Eve and Dirk Bucher (2001). “Central pattern generators and the control of rhythmic movements”. In: Current biology. Merel, Josh, Matthew Botvinick, and Greg Wayne (2019). “Hierarchical motor control in mammals and machines”. In: Nature Communications 10.1, pp. 1–12. Mnih, Volodymyr et al. (2015). “Human-level control through deep reinforcement learning”. In: Nature 518.7540, p. 529. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Shalit, Uri et al. (2012). “Descending systems translate transient cortical commands into a sustained muscle activation signal”. In: Cerebral cortex 22.8, pp. 1904–1914. 123 Tassa, Yuval et al. (2020). “dm_control: Software and tasks for continuous control”. In: arXiv preprint arXiv:2006.12983. Ziebart, Brian D (2010). “Modeling purposeful adaptive behavior with the principle of maximum causal entropy”. PhD thesis. CMU. Part IV Discussion 124 125 Chapter 8 CONNECTIONS TO PREDICTIVE CODING & NEUROSCIENCE Marino, Joseph (2019). “Predictive Coding, Variational Autoencoders, and Biological Connections”. In: NeurIPS Workshop on Real Neurons and Hidden Units. url: https://openreview.net/forum?id=SyeumQYUUH. 8.1 Introduction The previous chapters brought feedback and feedforward ideas from predictive coding into the realm of machine learning. In this chapter, we more explicitly identify these connections, implying surprising analogies between machine learning and neuroscience. We start in Section 8.2 by describing predictive coding, its hypothesized connections to neuroscience, and the empirical evidence for this overall theory. In Section 8.3, we then connect the ideas developed in this thesis back to their origins in predictive coding. By connecting these areas through this conceptual bridge, we arrive at new perspectives on the possible correspondences between machine learning and neuroscience (Section 8.4). These correspondences draw into question fundamental assumptions on the analogy of biological and artificial neurons (McCulloch and Pitts, 1943), with a host of implications for learning, inference, inhibition, etc. Unlike the technical contributions in Parts II & III, this chapter is more speculative, meant to serve as a starting point for further cross-pollination between machine learning and neuroscience. Like the works of Broeke, 2016 and Lotter, Kreiman, and Cox, 2018, we hope that these ideas will inspire future research in exploring this promising paradigm. 8.2 Predictive Coding Predictive coding, as described within neuroscience, can be divided into two separate settings, spatiotemporal and hierarchical, corresponding to the two main forms of structured probabilistic dependencies (Chapter 2). We review each of these settings, discussing previously hypothesized correspondences with neural anatomy. Finally, we outline the current empirical support for predictive coding in neural systems, highlighting the need for large-scale, testable models. 126 Spatiotemporal Predictive Coding Spatiotemporal predictive coding (Srinivasan, Laughlin, and Dubs, 1982), as the name implies, involves forming predictions across spatial dimensions and temporal sequences. These predictions then produce the resulting “code” as the prediction error. Concretely, in the temporal setting, we can consider a Gaussian autoregressive model, 𝑝 𝜃 , defined over observation sequences, x1:𝑇 . The conditional probability at time 𝑡 can be written as 𝑝 𝜃 (x𝑡 |x<𝑡 ) = N (x𝑡 ; µ𝜃 (x<𝑡 ), diag(σ𝜃2 (x<𝑡 ))). Introducing auxiliary variables, y𝑡 ∼ N (0, I), we can use the reparameterization trick to express x𝑡 = µ𝜃 (x<𝑡 ) + σ𝜃 (x<𝑡 ) y𝑡 , where denotes element-wise multiplication. Conversely, we can express the inverse, normalization or whitening transform as y𝑡 = x𝑡 − µ𝜃 (x<𝑡 ) . σ𝜃 (x<𝑡 ) (8.1) An example of temporal normalization with video, adapted from Chapter 5, is shown in Figure 8.1b. Note that one special case of this transform involves setting µ𝜃 (x<𝑡 ) ≡ x𝑡−1 and σ𝜃 (x<𝑡 ) ≡ 1, in which case, y𝑡 = x𝑡 − x𝑡−1 , i.e. temporal differences. For sequences that change slowly relative to the temporal step-size, this is a reasonable assumption. As noted in Chapter 5, this inverse transform can remove temporal redundancy in the input sequence. Thus, by Shannon’s source coding theorem (Shannon, 1948), we can encode or compress y1:𝑇 more efficiently than x1:𝑇 . The benefit of this sequential predictive coding scheme was recognized in the early days of information theory (Harrison, 1952; Oliver, 1952), forming the basis of modern video (Wiegand et al., 2003) and audio (Atal and Schroeder, 1979) compression. A similar process can also be applied within x𝑡 to remove spatial dependencies. For instance, we could also apply an autoregressive affine transform over spatial dimensions, predicting the 𝑖 th dimension, 𝑥𝑖,𝑡 , as a function of previous spatial dimensions, x1:𝑖,𝑡 . With linear functions, this corresponds to Cholesky whitening (Pourahmadi, 2011; Kingma, Salimans, et al., 2016). However, this requires imposing an arbitrary ordering over spatial dimensions. Perhaps a more reasonable approach in the spatial setting is to learn a set of symmetric dependencies between dimensions. Here, the linear case corresponds to ZCA whitening (Kessy, Lewin, and Strimmer, 2018), shown in Figure 8.1a. In the natural image domain, both of these whitening schemes generally result in center-surround spatial filters, extracting edges from the input. 127 yt <latexit sha1_base64="Rc8skmreVThEy0HMt+P8j5ZBRTE=">AAACgXicdVHLSgMxFE2nPmp9tbp0EywF3ZSZKii4KbpxqWBV6AxDJs20ockkJHfEMvQ33Opv+TdmahfW6oULh3Nfh3MTLbgF3/+seNW19Y3N2lZ9e2d3b7/RPHi0KjeU9akSyjwnxDLBM9YHDoI9a8OITAR7SiY3Zf3phRnLVfYAU80iSUYZTzkl4KgwlATGSVpMZzHEjZbf8eeBV0GwAC20iLu4WYnDoaK5ZBlQQawdBL6GqCAGOBVsVg9zyzShEzJiAwczIpmNirnoGW47ZohTZVxmgOfsz4mCSGunMnGdpUj7u1aSf9UGOaSXUcEznQPL6PehNBcYFC4dwENuGAUxdYBQw51WTMfEEArOp6VND0FUlOLKNUvnEzmrt/FPppShQb4u9xExUu7CWP5Dc7o6oC3Lnatq6Ax0Lwl+P2AVPHY7wVmne3/e6l0vnlNDR+gYnaAAXaAeukV3qI8o0ugNvaMPr+qder7X/W71KouZQ7QU3tUXUArHDg==</latexit> <latexit sha1_base64="tuXXMIg/Bc4YouWi1xyioUAsaLE=">AAACm3icdVHLTttAFJ24tIX0QaDLqtKIFIluIhuQ2gULBJuqYkElAkixZV1PrpMRMx5r5roisvwD/Rq27Z/0bzoOWRACVxrp6NzXmXOzUklHYfivE7xYe/nq9fpG983bd+83e1vbl85UVuBQGGXsdQYOlSxwSJIUXpcWQWcKr7Kb0zZ/9Qutk6a4oFmJiYZJIXMpgDyV9j7Hma5jJycamjSmKRLwvVgDTbO8vm3S+oiaL2mvHw7CefBVEC1Any3iPN3qpPHYiEpjQUKBc6MoLCmpwZIUCptuXDksQdzABEceFqDRJfX8Ow3f9cyY58b6VxCfsw87atDOzXTmK1uh7nGuJZ/KjSrKvyW1LMqKsBD3i/JKcTK89YaPpUVBauYBCCu9Vi6mYEGQd3Bp0kWU1K24dszS+kw33V3+kGlllKRvl+tATYzfMNXP0FKsNpQOK++qGXsD/UmixwdYBZf7g+hgsP/zsH98sjjOOvvIdtgei9hXdsy+s3M2ZIL9ZnfsD/sbfApOgx/B2X1p0Fn0fGBLEQz/A+6F0TQ=</latexit> ⇤ ✓ (x<t ) ÷ <latexit sha1_base64="zU1t23bQbWEUhSwGFZ+dhUDLseE=">AAACeXicdVHLTgIxFO2ML8QX6NJNlZAQF2QGTHRJdOMSE14JTEin04HGdjppO0Qy4Rfc6q/5LW7sAAsG9CZNTs59nNN7/ZhRpR3n27L39g8OjwrHxZPTs/OLUvmyp0QiMeliwYQc+EgRRiPS1VQzMoglQdxnpO+/PWf5/oxIRUXU0fOYeBxNIhpSjHRGjQI6G5cqTt1ZBtwF7hpUwDra47I1HgUCJ5xEGjOk1NB1Yu2lSGqKGVkUR4kiMcJvaEKGBkaIE+WlS7MLWDVMAEMhzYs0XLKbHSniSs25byo50lO1ncvIv3LDRIePXkqjONEkwiuhMGFQC5j9HAZUEqzZ3ACEJTVeIZ4iibA2+8lN6rhempnLxuTkfb4oVuEmk9mINX/P1yE2EUZhyv+hKd5tiBVJzFZFYBZoTuJuH2AX9Bp1t1lvvN5XWk/r4xTANbgFNeCCB9ACL6ANugCDKfgAn+DL+rFv7Jp9tyq1rXXPFciF3fwFDZDESg==</latexit> ✓ <latexit sha1_base64="2DLctlRnln8rUGvY6ScommwatuM=">AAACe3icdVHLSgMxFE3HV62vVpdugqUgRcpMVXRZdOOyQl/QDiWTpm00mYQkI5Zh/sGt/pkfI5hpZ9GHXggczn2ck3sDyag2rvudc7a2d3b38vuFg8Oj45Ni6bSjRaQwaWPBhOoFSBNGQ9I21DDSk4ogHjDSDV4f03z3jShNRdgyM0l8jiYhHVOMjKU6g4DH1WRYLLs1dx5wE3gZKIMsmsNSbjgYCRxxEhrMkNZ9z5XGj5EyFDOSFAaRJhLhVzQhfQtDxIn247ndBFYsM4JjoewLDZyzyx0x4lrPeGArOTJTvZ5Lyb9y/ciM7/2YhjIyJMQLoXHEoBEw/TscUUWwYTMLEFbUeoV4ihTCxm5oZVLL8+PUXDpmRT7gSaECl5nUhjT8fbUOsYmwClP+D03xZoPUJLJbFSO7QHsSb/0Am6BTr3nXtfrzTbnxkB0nD87BBbgEHrgDDfAEmqANMHgBH+ATfOV+nLJTda4WpU4u6zkDK+Hc/gLEQ8UM</latexit> <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="YlEnZegu6BMRciM97uIB7y4yLos=">AAACdnicdVHLTgIxFC3jC/EFujQxjYToRjKDJrokunEJCa8EJqRTLtDQTidtx0gmfIFb/Tj/xKUdYMFDb9Lk5NzHOb03iDjTxnW/M87O7t7+QfYwd3R8cnqWL5y3tIwVhSaVXKpOQDRwFkLTMMOhEykgIuDQDiYvab79BkozGTbMNAJfkFHIhowSY6n6XT9fdMvuPPA28JagiJZR6xcy/d5A0lhAaCgnWnc9NzJ+QpRhlMMs14s1RIROyAi6FoZEgPaTudMZLllmgIdS2RcaPGdXOxIitJ6KwFYKYsZ6M5eSf+W6sRk++QkLo9hASBdCw5hjI3H6bTxgCqjhUwsIVcx6xXRMFKHGLmdtUsPzk9RcOmZNPhCzXAmvMqmNyIj39TrCR9IqjMU/NKPbDZGG2G5VDuwC7Um8zQNsg1al7N2XK/WHYvV5eZwsukTX6BZ56BFV0SuqoSaiCNAH+kRfmR/nyik5N4tSJ7PsuUBr4bi/nK/Cug==</latexit> <latexit sha1_base64="YlEnZegu6BMRciM97uIB7y4yLos=">AAACdnicdVHLTgIxFC3jC/EFujQxjYToRjKDJrokunEJCa8EJqRTLtDQTidtx0gmfIFb/Tj/xKUdYMFDb9Lk5NzHOb03iDjTxnW/M87O7t7+QfYwd3R8cnqWL5y3tIwVhSaVXKpOQDRwFkLTMMOhEykgIuDQDiYvab79BkozGTbMNAJfkFHIhowSY6n6XT9fdMvuPPA28JagiJZR6xcy/d5A0lhAaCgnWnc9NzJ+QpRhlMMs14s1RIROyAi6FoZEgPaTudMZLllmgIdS2RcaPGdXOxIitJ6KwFYKYsZ6M5eSf+W6sRk++QkLo9hASBdCw5hjI3H6bTxgCqjhUwsIVcx6xXRMFKHGLmdtUsPzk9RcOmZNPhCzXAmvMqmNyIj39TrCR9IqjMU/NKPbDZGG2G5VDuwC7Um8zQNsg1al7N2XK/WHYvV5eZwsukTX6BZ56BFV0SuqoSaiCNAH+kRfmR/nyik5N4tSJ7PsuUBr4bi/nK/Cug==</latexit> µ✓ (x<t ) <latexit sha1_base64="wy1WYPqqnkkv22S/qgVxrO+jOM4=">AAACmHicdVFNT9tAEN24tNC0lAC39rJthEQvkQ1I9MAhggP0BogAUmxZ6804WbHrtXbHVSLL5/4arvBb+DeskxwIgZFWenrz9fZNkkth0fefGt6HlY+fVtc+N798Xf+20drcura6MBx6XEttbhNmQYoMeihQwm1ugKlEwk1yd1Lnb/6BsUJnVzjJIVJsmIlUcIaOils/w0SVoSqqOMQRIKO7oWI4StJyXMXlEVa/41bb7/jToMsgmIM2mcd5vNmIw4HmhYIMuWTW9gM/x6hkBgWXUDXDwkLO+B0bQt/BjCmwUTn9S0V3HDOgqTbuZUin7MuOkilrJypxlbVQ+zpXk2/l+gWmf6JSZHmBkPHZorSQFDWtjaEDYYCjnDjAuBFOK+UjZhhHZ9/CpKsgKmtx9ZiF9Ymqmjv0JVPLyFGNF+uYHGq3YaTeoQVfbsgtFM5VPXAGupMErw+wDK73OsF+Z+/ioN09nh9njfwgv8guCcgh6ZIzck56hJP/5J48kEfvu9f1Tr2/s1KvMe/ZJgvhXT4D4PDP5w==</latexit> xt 3 <latexit sha1_base64="A+NwkXFLy8dEXhO4lC/Y2RBrA7Y=">AAACh3icdVHLSsNAFJ3GZ+sr6tLNYBHcWJMq2qWPjUsFq0IbwmQ6aQdnMmHmRlpC/sSt/pN/46Ttwli9cOFw7utwbpQKbsDzvmrO0vLK6tp6vbGxubW94+7uPRmVacq6VAmlXyJimOAJ6wIHwV5SzYiMBHuOXm/L+vMb04ar5BEmKQskGSY85pSApULX7UsCoyjOx0WYw8lZEbpNr+VNAy8Cfw6aaB734W4t7A8UzSRLgApiTM/3UghyooFTwYpGPzMsJfSVDFnPwoRIZoJ8Kr3AR5YZ4FhpmwngKftzIifSmImMbGcp1PyuleRftV4GcSfIeZJmwBI6OxRnAoPCpQ94wDWjICYWEKq51YrpiGhCwbpV2fToB3kprlxTOR/JonGEfzKljBTkuNpHxFDZCyP5D83p4kBqWGZdVQNroH2J//sBi+Cp3fLPWu2H8+bVzfw56+gAHaJj5KNLdIXu0D3qIore0Dv6QJ9O3Tl1LpzOrNWpzWf2USWc629L3Mi+</latexit> (a) Spatial xt 2 <latexit sha1_base64="FQVlMEFQvZFJH0/AEMcNlsO9I7E=">AAACh3icdVHLTsJAFB3qC/AFunQzkZC4EVs06tLHxiUmoibQNNNhChNnOs3MrYE0/RO3+k/+jVNgIaA3ucnJua+Tc8NEcAOu+11y1tY3NrfKler2zu7efq1+8GxUqinrUiWUfg2JYYLHrAscBHtNNCMyFOwlfLsv6i/vTBuu4ieYJMyXZBjziFMClgpqtb4kMAqjbJwHGZy286DWcFvuNPAq8OaggebRCeqloD9QNJUsBiqIMT3PTcDPiAZOBcur/dSwhNA3MmQ9C2MimfGzqfQcNy0zwJHSNmPAU/b3REakMRMZ2s5CqFmuFeRftV4K0bWf8ThJgcV0dihKBQaFCx/wgGtGQUwsIFRzqxXTEdGEgnVrYdOT52eFuGLNwvlQ5tUm/s0UMhKQ48U+IobKXhjJf2hOVwcSw1LrqhpYA+1LvOUHrILndss7b7UfLxo3d/PnlNEROkYnyENX6AY9oA7qIore0Qf6RF9OxTlzLp3rWatTms8cooVwbn8AScnIvQ==</latexit> xt 1 <latexit sha1_base64="Uoe094Ebd5E0MjBKCabrmYfr41Y=">AAACh3icdVHJTsMwEHXD1patwJGLRYXEhZIUBBxZLhxB6ia1UeS4Tmthx5E9qaii/AlX+Cf+Bqf00FIYaaSnN9vTmzAR3IDrfpWctfWNza1ypbq9s7u3Xzs47BiVasraVAmleyExTPCYtYGDYL1EMyJDwbrh62NR706YNlzFLZgmzJdkFPOIUwKWCmq1gSQwDqPsLQ8yOPfyoFZ3G+4s8Crw5qCO5vEcHJSCwVDRVLIYqCDG9D03AT8jGjgVLK8OUsMSQl/JiPUtjIlkxs9m0nN8apkhjpS2GQOesYsTGZHGTGVoOwuh5netIP+q9VOIbv2Mx0kKLKY/h6JUYFC48AEPuWYUxNQCQjW3WjEdE00oWLeWNrU8PyvEFWuWzocyr57iRaaQkYB8W+4jYqTshbH8h+Z0dSAxLLWuqqE10L7E+/2AVdBpNrzLRvPlqn73MH9OGR2jE3SGPHSD7tATekZtRNEEvaMP9OlUnAvn2rn9aXVK85kjtBTO/TdHtsi8</latexit> xt <latexit sha1_base64="/9eKoYxPE1ZizRUsfEHodALxXRM=">AAACg3icdVHLSgMxFE1H66O+Wl26CZaCIJSZWtCNILpxqdAXtGPJpJk2mEyG5I60DPMfbvWv/BsztYs+9MKFw7mvw7lBLLgB1/0uOFvbxZ3dvf3SweHR8Um5ctoxKtGUtakSSvcCYpjgEWsDB8F6sWZEBoJ1g7fHvN59Z9pwFbVgFjNfknHEQ04JWOp1IAlMgjCdZsMUsmG56tbdeeBN4C1AFS3ieVgpDAcjRRPJIqCCGNP33Bj8lGjgVLCsNEgMiwl9I2PWtzAikhk/ncvOcM0yIxwqbTMCPGeXJ1IijZnJwHbmMs16LSf/qvUTCG/9lEdxAiyiv4fCRGBQOPcAj7hmFMTMAkI1t1oxnRBNKFinVja1PD/NxeVrVs4HMivV8DKTy4hBTlf7iBgre2Ei/6E53RyIDUusq2pkDbQv8dYfsAk6jbp3XW+8NKv3D4vn7KFzdIEukYdu0D16Qs+ojSjS6AN9oi+n6Fw5Daf52+oUFjNnaCWcux+lOcgZ</latexit> (b) Temporal 35 31 Figure 8.1: Spatiotemporal Predictive Coding. (a) Spatial predictive coding models and removes spatial dependencies. In the domain of natural images, one version of linear predictive coding is ZCA whitening, which yields center-surround filters (left). As a result, the whitened image contains highlighted edges (right). (b) Temporal predictive coding models and removes temporal dependencies. In the domain of natural video, this tends to remove static backgrounds. Srinivasan, Laughlin, and Dubs, 1982 investigated the principles of spatiotemporal predictive coding in the retina, where compression is essential for transmission through the optic nerve. Estimating the auto-correlation function of input sensory signals, i.e. a linear prediction, they showed that spatiotemporal predictive coding provides a reasonable fit to retinal ganglion cell recordings from flies, allowing the retina’s output neurons to more fully utilize their dynamic range. It is now generally accepted that retina, in part, performs stages of spatial and temporal normalization through center-surround receptive fields and on-off responses (Hosoya, Baccus, and Meister, 2005; Graham, Chandler, and Field, 2006; Pitkow and Meister, 2012; Palmer et al., 2015). Dong and Atick, 1995 applied similar predictive coding ideas to the thalamus, proposing an additional stage of temporal normalization. Likewise, Friston’s use of generalized coordinates (K. Friston, 2008a), i.e. modeling multiple orders of temporal derivatives, can be approximated using finite temporal differences through repeated application of predictive coding. That is, 𝑑x 𝑑𝑡 ≈ Δx𝑡 ≡ x𝑡 − x𝑡−1 . Thus, spatiotemporal predictive coding may be utilized at multiple stages of sensory processing to remove redundancy (Y. Huang and Rao, 2011). In neural circuits, spatiotemporal normalization often involves inhibitory interneurons (Carandini and Heeger, 2012), carrying out operations similar to those in Eq. 8.1 (though other mechanisms are also possible). For instance, retinal inhibitory interactions take place between photoreceptors, via horizontal cells, and between 128 bipolar cells, via amacrine cells. This enables unpredicted motion to be computed, e.g. an object moving relative to the background, using inhibitory interactions from amacrine cells (Ölveczky, Baccus, and Meister, 2003; Baccus et al., 2008). Similar inhibitory interactions are present in the lateral geniculate nucleus (LGN) in thalamus, with interneurons inhibiting relay cells originating from retina (Sherman and Guillery, 2002). As mentioned above, this is thought to implement a form of temporal normalization (Dong and Atick, 1995), removing, at least, linear dependencies (Dan, Atick, and Reid, 1996). Inhibition via lateral inhibitory interactions is also a prominent feature of neocortex, with distinct classes of local interneurons playing a significant role in shaping the responses of principal pyramidal neurons (Isaacson and Scanziani, 2011). While these distinct classes may serve separate computational roles, part of this purpose appears to be for spatiotemporal normalization (Carandini and Heeger, 2012). Finally, while we have focused largely on early stages of sensory processing, inhibitory interneurons are also prevalent in other areas of neocortex, as well as in central pattern generator (CPG) circuits (Marder and Bucher, 2001), found in the spinal cord. These circuits are repsonsible for the rhythmic generation of movement, such as locomotion. Thus, just as inhibitory interactions remove spatiotemporal dependencies in early sensory areas, similar computational operations can add spatiotemporal dependencies in motor activation. Hierarchical Predictive Coding The other main form of predictive coding, mathematically formulated by Rao and Ballard, 1999; K. Friston, 2005, involves hierarchies of latent variables and, as such, has been postulated as a model of hierarchical cortical processing. The neocortex (Figure 8.2) is a sheet-like structure involved in many aspects of sensory and motor processing. It is composed of six layers (I–VI), containing particular classes of neurons and connections. Across layers, neurons are arranged into columns, which are engaged in related computations (Mountcastle, Berman, and Davies, 1955). Columns interact locally via inhibitory interactions from interneurons while also forming processing hierarchies through longer-range excitatory interactions from pyramidal neurons. Particularly in earlier sensory areas, longer-range connections are generally grouped into forward (up the hierarchy) and backward (down the hierarchy) directions. Forward connections are traditionally thought to be driving (evoking neural activity) (Girard and Bullier, 1989; Girard, Salin, and Bullier, 1991). Backward connections are traditionally thought to be modulatory, however, they have also been shown be driving (Covic and Sherman, 2011; De Pasquale and Sherman, 129 backward <latexit sha1_base64="qKHL+et+VPCk7wZOifLgtLKiksM=">AAACiHicdVHLSgMxFE3HV62vVpdugqXgqsyooO6kblwq2Ae0Q8lkbttgMhmSO2oZ+ilu9Zv8GzO1C2v1woXDua/DuVEqhUXf/yx5a+sbm1vl7crO7t7+QbV22LE6MxzaXEttehGzIEUCbRQooZcaYCqS0I2ebot69xmMFTp5xGkKoWLjRIwEZ+ioYbU2QHhFo/KI8acXZuLZsFr3m/486CoIFqBOFnE/rJWGg1jzTEGCXDJr+4GfYpgzg4JLmFUGmYXUrWdj6DuYMAU2zOfaZ7ThmJiOtHGZIJ2zPydypqydqsh1KoYT+7tWkH/V+hmOrsJcJGmGkPDvQ6NMUtS0MILGwgBHOXWAcSOcVsonzDCOzq6lTY9BmBfiijVL5yM1qzToT6aQkaJ6Xe5jcqzdhYn6hxZ8dSC1kDlXdewMdC8Jfj9gFXTOmsF58+zhon7TWjynTI7JCTklAbkkN+SO3JM24eSFvJF38uFVPN+79K6/W73SYuaILIXX+gIkSsmW</latexit> neocortex <latexit sha1_base64="Cw2niyy9AT4DZKbyux4FTcgsTlo=">AAACiXicdVHLTgIxFC3jG1+oSzeNxMQVmVETiSujG5eaCJjAhHTKBRr7mLR3DGTCr7jVX/Jv7AALEL1Jk9NzXyfnJqkUDsPwuxSsrW9sbm3vlHf39g8OK0fHTWcyy6HBjTT2NWEOpNDQQIESXlMLTCUSWsnbQ5FvvYN1wugXHKcQKzbQoi84Q091K8cdhBFalWsw3Fj/mXQr1bAWToOugmgOqmQeT92jUrfTMzxToJFL5lw7ClOMc2ZRcAmTcidzkDL+xgbQ9lAzBS7Op+In9NwzPdo31j+NdMouduRMOTdWia9UDIfud64g/8q1M+zX41zoNEPQfLaon0mKhhZO0J6wwFGOPWDcCq+V8iGzjKP3a2nSSxTnhbhizNL6RE3K53SRKWSkqEbLdUwOjN8wVP/Qgq82pA4y76rpeQP9SaLfB1gFzctadFW7fL6u3t3Pj7NNTskZuSARuSF35JE8kQbhZEQ+yCf5CnaDKKgHt7PSoDTvOSFLETz8AIzAyjg=</latexit> forward <latexit sha1_base64="T5QNpCX2RkA+7wVN835MVF2ZG+w=">AAACh3icdVFNTwIxEC3rN36BHr00EhNPuKtGPaJePGoiaAIb0u0O0NhuN+2sQjb8E6/6n/w3dpGDgE7S5OXNTN/LmyiVwqLvf5W8peWV1bX1jfLm1vbObqW617I6MxyaXEttniNmQYoEmihQwnNqgKlIwlP0clv0n17BWKGTRxylECrWT0RPcIaO6lYqHYQhGpX3tHljJh53KzW/7k+KLoJgCmpkWvfdaqnbiTXPFCTIJbO2HfgphjkzKLiEcbmTWUgZf2F9aDuYMAU2zCfWx/TIMTF14u4lSCfs742cKWtHKnKTiuHAzvcK8q9eO8PeVZiLJM0QEv4j1MskRU2LHGgsDHCUIwcYN8J5pXzADOPo0pr56TEI88Jc8c2MfKTG5SP6mylspKiGs3NM9rVTGKh/aMEXF1ILmUtVxy5Ad5Jg/gCLoHVaD87qpw/ntcbN9Djr5IAckmMSkEvSIHfknjQJJ6/knXyQT2/DO/EuvKufUa803dknM+VdfwNfW8lC</latexit> I <latexit sha1_base64="Zdhq5wJxAidLE41yxJeKnvaUBsw=">AAACf3icdVHLTgIxFC3jG1+gSzeNxOiKzKCJujO60Z0mokaYkE65QEMfk/aOkUz4C7f6X/6NHWQhojdpcnLu45zem6RSOAzDz1KwsLi0vLK6Vl7f2NzarlR3HpzJLIcmN9LYp4Q5kEJDEwVKeEotMJVIeEyGV0X+8QWsE0bf4yiFWLG+Fj3BGXrquY3wilblN+NOpRbWw0nQeRBNQY1M47ZTLXXaXcMzBRq5ZM61ojDFOGcWBZcwLrczBynjQ9aHloeaKXBxPrE8pgee6dKesf5ppBP2Z0fOlHMjlfhKxXDgfucK8q9cK8PeWZwLnWYImn8L9TJJ0dDi/7QrLHCUIw8Yt8J7pXzALOPotzQz6T6K88JcMWZGPlHj8gH9yRQ2UlSvs3VM9o1XGKh/aMHnG1IHmd+q6foF+pNEvw8wDx4a9ei43rg7qV1cTo+zSvbIPjkiETklF+Sa3JIm4USTN/JOPoJScBjUg/C7NChNe3bJTATnX0wgxik=</latexit> II/III <latexit sha1_base64="W+jyTp8xBeQu16OU85Ur4jl+6R4=">AAAChnicdVHLTgIxFC3jC/EB6tJNIzFxhTM+okuiG2eniagJTEinXKCxnU7aOwYy4Uvc6kf5N3aAhYje5CYn575Ozo1TKSz6/lfJW1ldW98ob1a2tnd2q7W9/SerM8OhxbXU5iVmFqRIoIUCJbykBpiKJTzHr7dF/fkNjBU6ecRxCpFig0T0BWfoqG6t2kEYoVF5GJ6GYTjp1up+w58GXQbBHNTJPO67e6Vup6d5piBBLpm17cBPMcqZQcElTCqdzELK+CsbQNvBhCmwUT5VPqHHjunRvjYuE6RT9udEzpS1YxW7TsVwaH/XCvKvWjvD/nWUiyTNEBI+O9TPJEVNCxtoTxjgKMcOMG6E00r5kBnG0Zm1sOkxiPJCXLFm4XysJpVj+pMpZKSoRot9TA60uzBU/9CCLw+kFjLnqu45A91Lgt8PWAZPZ43gvHH2cFFv3syfUyaH5IickIBckSa5I/ekRTjJyDv5IJ9e2Wt4l97VrNUrzWcOyEJ4zW9ldMff</latexit> sensory <latexit sha1_base64="ibmQ59SNdiWWlf3YzZ3i5JjErig=">AAACh3icdVHJTgJBEG3GHTfUo5eOxMQTzqhRjy4Xj5qAkMCE9DQFdOxl0l1DJBP+xKv+k39jD3IQ0UoqeXm1vbxKUikchuFnKVhaXlldW98ob25t7+xW9vafnckshwY30thWwhxIoaGBAiW0UgtMJRKayct9UW+OwDphdB3HKcSKDbToC87QU91KpYPwilblDrQzdjzpVqphLZwGXQTRDFTJLB67e6Vup2d4pkAjl8y5dhSmGOfMouASJuVO5iBl/IUNoO2hZgpcnE+lT+ixZ3q0b6xPjXTK/pzImXJurBLfqRgO3e9aQf5Va2fYv45zodMMQfPvQ/1MUjS08IH2hAWOcuwB41Z4rZQPmWUcvVtzm+pRnBfiijVz5xM1KR/Tn0whI0X1Ot/H5MD4C0P1Dy344kDqIPOump430L8k+v2ARfB8VovOa2dPF9Wbu9lz1skhOSInJCJX5IY8kEfSIJyMyBt5Jx/BRnAaXAbX361BaTZzQOYiuP0CnbHJYA==</latexit> IV <latexit sha1_base64="YDXl5+A1g/Db4FDKg/rZ1qK+jiU=">AAACgHicdVHLTgIxFC3jG1+gSzeNhIQVzqiJxhXRje4wETSBCemUCzS200l7x0AmfIZb/S7/xg6yANGb3OTk3NfJuVEihUXf/yp4a+sbm1vbO8Xdvf2Dw1L5qG11aji0uJbavETMghQxtFCghJfEAFORhOfo9S6vP7+BsULHTzhJIFRsGIuB4Awd1ekijNGo7KE97ZUqft2fBV0FwRxUyDyavXKh1+1rniqIkUtmbSfwEwwzZlBwCdNiN7WQMP7KhtBxMGYKbJjNNE9p1TF9OtDGZYx0xi5OZExZO1GR61QMR/Z3LSf/qnVSHFyHmYiTFCHmP4cGqaSoaW4A7QsDHOXEAcaNcFopHzHDODqbljY9BWGWi8vXLJ2P1LRYpYtMLiNBNV7uY3Ko3YWR+ocWfHUgsZA6V3XfGeheEvx+wCpon9eDi/r542WlcTt/zjY5IaekRgJyRRrknjRJi3CiyTv5IJ+e59W8My/4afUK85ljshTezTcn2caJ</latexit> input <latexit sha1_base64="20mWnOjt/2TLxGcCMAx5q9rOGtg=">AAACgnicdVHLTgIxFC0jvvAFunTTSEyMC5xRE124ILrRnSaAJjAhnXKBxnY6tncMZMJ3uNXP8m/sIAsRvclNTs59nZwbJVJY9P3PgrdUXF5ZXVsvbWxube+UK7stq1PDocm11OYpYhakiKGJAiU8JQaYiiQ8Rs83ef3xFYwVOm7gOIFQsUEs+oIzdFTYQRihUVnrpHU36Zarfs2fBl0EwQxUySzuu5VCt9PTPFUQI5fM2nbgJxhmzKDgEialTmohYfyZDaDtYMwU2DCbqp7QQ8f0aF8blzHSKftzImPK2rGKXKdiOLS/azn5V62dYv8yzEScpAgx/z7UTyVFTXMLaE8Y4CjHDjBuhNNK+ZAZxtEZNbepEYRZLi5fM3c+UpPSIf3J5DISVKP5PiYH2l0Yqn9owRcHEgupc1X3nIHuJcHvByyC1mktOKudPpxX69ez56yRfXJAjkhALkid3JJ70iScvJA38k4+vKJ37AXe2XerV5jN7JG58K6+AI84xyI=</latexit> V/VI <latexit sha1_base64="gkcS4I2+2kbbiuHa23v+P/BzYOc=">AAACg3icdVHLSgMxFE3Hd33r0k2wCIJQZmpBN0LRjUsF+4B2LJn0tg0mkyG5Iy1D/8Ot/pV/Y6btwrZ6IXA493VybpRIYdH3vwve2vrG5tb2TnF3b//g8Oj4pGF1ajjUuZbatCJmQYoY6ihQQisxwFQkoRm9PeT55jsYK3T8guMEQsUGsegLztBRrx2EERqViThJcdI9Kvllfxp0FQRzUCLzeOoeF7qdnuapghi5ZNa2Az/BMGMGBZcwKXZSCwnjb2wAbQdjpsCG2VT2hF44pkf72rgXI52yvzsypqwdq8hVKoZDu5zLyb9y7RT7t+HsTxDz2aJ+KilqmntAe8IARzl2gHEjnFbKh8wwjs6phUkvQZjl4vIxC+sjNSle0N9MLiNBNVqsY3Kg3Yah+ocWfLUhsZA6V3XPGehOEiwfYBU0KuXgulx5rpZq9/PjbJMzck4uSUBuSI08kidSJ5wY8kE+yZe34V15Fa86K/UK855TshDe3Q/mzMg4</latexit> thalamus <latexit sha1_base64="EVQUosQp+z/R/w+EWWvxg721d/s=">AAACiHicdVHLSgNBEJys7/hK9OhlMAiewq4K6k304lHBqJAsoXfSSQZndpaZXjUs+RSv+k3+jbMxB2O0oaGofhXVSaakozD8rAQLi0vLK6tr1fWNza3tWn3n3pncCmwJo4x9TMChkim2SJLCx8wi6EThQ/J0VdYfntE6adI7GmUYaxiksi8FkKe6tXqH8JWsLmgICnTuxt1aI2yGk+DzIJqCBpvGTbde6XZ6RuQaUxIKnGtHYUZxAZakUDiudnKHGYgnGGDbwxQ0uriYaB/zA8/0eN9YnynxCftzogDt3EgnvlMDDd3vWkn+VWvn1D+LC5lmOWEqvg/1c8XJ8NII3pMWBamRByCs9Fq5GIIFQd6umU13UVyU4so1M+cTPa4e8J9MKSMj/TrbB2pg/IWh/oeWYn4gc5h7V03PG+hfEv1+wDy4P2pGx82j25PGxeX0Oatsj+2zQxaxU3bBrtkNazHBXtgbe2cfQTUIg9Pg/Ls1qExndtlMBJdfZyXJtg==</latexit> neocortical microcircuit <latexit sha1_base64="Bg1O/K1VDFmiCK1AZXtAm7J/Rq8=">AAACi3icdVHJSgNBEO2Me9zicvPSGARPYUYFRTyIIniMYFRIhtDTqSSNvQzdNWIc8i9e9Y/8G3tiDonRgoLHe7U8qpJUCodh+FUK5uYXFpeWV8qra+sbm5Wt7QdnMsuhwY009ilhDqTQ0ECBEp5SC0wlEh6T5+tCf3wB64TR9zhIIVasp0VXcIaeald2WwivaFWuwXBj0Qty2K5Uw1o4CjoLojGoknHU21uldqtjeKZAI5fMuWYUphjnrBgoYVhuZQ5Sxp9ZD5oeaqbAxfnI/pAeeKZDu8b61EhH7GRHzpRzA5X4SsWw735rBfmX1sywexbnQqcZguY/i7qZpGhocQvaERY4yoEHjFvhvVLeZ5Zx9BebmnQfxXlhrhgztT5Rw/IBnWQKGymq1+k6JnvGb+irf2jBZxtSB5m/qun4A/qXRL8fMAsejmrRce3o7qR6eTV+zjLZI/vkkETklFySW1InDcLJG3knH+QzWA+Og/Pg4qc0KI17dshUBDffbSHLCA==</latexit> <latexit sha1_base64="27bCNqqVu8xZZ4yc90d2CNoBW+Y=">AAACjHicdVFdSxtBFJ2sbY1paxPFJ1+GBqFPYdcKCiIEheKjgjGBZAmzNzfJ4MzOMnNXEpb8GF/1F/lvOhvzYEx74cLh3K/DuUmmpKMwfK0EW58+f9mu7tS+fvu++6Pe2Lt3JreAHTDK2F4iHCqZYockKexlFoVOFHaTh6uy3n1E66RJ72ieYazFJJVjCYI8NawfDAhnZHWhJVgD0kIuaTGsN8NWuAy+CaIVaLJV3AwbleFgZCDXmBIo4Vw/CjOKC2FJgsJFbZA7zAQ8iAn2PUyFRhcXS/0LfuSZER8b6zMlvmTfTxRCOzfXie/UgqbuY60k/1Xr5zQ+iwuZZjlhCm+HxrniZHhpBh9Ji0Bq7oEAK71WDlNhBZC3bG3TXRQXpbhyzdr5RC9qR/w9U8rISM/W+4SaGH9hqv9DS9gcyBzm3lUz8gb6l0QfH7AJ7o9b0e/W8e1Js325ek6VHbKf7BeL2Clrs2t2wzoMWMGe2DN7CXaDk+A8uHhrDSqrmX22FsGfv5fky4w=</latexit> Figure 8.2: Brain Anatomy & Cortical Circuitry. Left: Sensory inputs enter first-order relays in thalamus from sensory organs. Thalamus forms reciprocal connections with neocortex. Neocortex consists of hierarchies of cortical areas, with both forward and backward connections. Right: Neocortex is composed of six layers (I–VI), with specific neuron classes and connections at each layer. The simplified schematic depicts two cortical columns. Black and red circles represent excitatory and inhibitory neurons respectively, with arrows denoting major connections. This basic circuit motif is repeated with slight variations throughout neocortex. 2011), which can be inverted through inhibition (H. S. Meyer et al., 2011). These sets of connections, repeated with slight variations throughout neocortex, constitute a canonical neocortical microcircuit (Douglas, Martin, and Whitteridge, 1989), which could suggest a single processing algorithm (Hawkins and Blakeslee, 2004), capable of adapting to a variety of inputs (Sharma, Angelucci, and Sur, 2000). In formulating a theory of neocortex, Mumford, 1992 proposed that thalamus acts as an ‘active blackboard,’ with the cortical hierarchy attempting to reconstruct or predict the thalamic input and activity in areas throughout the hierarchy. Backward (topdown) projections would convey predictions, while forward (bottom-up) projections would use prediction errors to update the estimates throughout the hierarchy. Through a dynamic process of activation, the entire system would settle to a consistent pattern of activity, minimizing prediction error. Over longer periods of time, the model parameters would be adjusted to yield improved predictions. In this way, cortex would use negative feedback, both in inference and learning, to use and construct a generative model of its inputs. This notion of generative state estimation dates back (at least) to Helmholtz (Von Helmholtz, 1867), and the notion of correcting predictions based on prediction errors is inline with concepts from cybernetics (Wiener, 1948; MacKay, 1956), which influenced techniques like Kalman filtering (Kalman, 1960), a ubiquitous Bayesian filtering algorithm. A more complete mathematical formulation of this hierarchical predictive coding 130 model, with many similarities to Kalman filtering (see Rao, 1998), was provided by Rao and Ballard, 1999, with the generalization to variational inference provided by K. Friston, 2005. To illustrate this setup, consider a simple model consisting of a single level of continuous latent variables, z, modeling continuous data observations, x. We will use Gaussian densities for each distribution and assume we have 𝑝 𝜃 (x|z) = N (x; 𝑓 (Wz), diag(σx2 )), (8.2) 𝑝 𝜃 (z) = N (z; µz , diag(σz2 )), (8.3) where 𝑓 is an element-wise function (e.g. logistic sigmoid, tanh, or the identity), W is a weight matrix, µz is the constant prior mean, and σx2 and σz2 are constant vectors of variances. In the simplest approach to inference, we can find the maximum-a-posteriori (MAP) estimate, i.e. estimate the z∗ which maximizes 𝑝 𝜃 (z|x). While we cannot tractably evaluate 𝑝 𝜃 (z|x) directly, we can use Bayes’ rule to write z∗ = arg max 𝑝 𝜃 (z|x) z = arg max z 𝑝 𝜃 (x, z) 𝑝 𝜃 (x) = arg max 𝑝 𝜃 (x, z). z Thus, rather than evaluating the posterior distribution, 𝑝 𝜃 (z|x), we can perform this maximization using the joint distribution, 𝑝 𝜃 (x, z) = 𝑝 𝜃 (x|z) 𝑝 𝜃 (z), which we can tractably evaluate. We can also replace the optimization over the probability distribution with an optimization over the log probability, since log(·) is a monotonically increasing function and will not affect the optimization. We then have z∗ = arg max [log 𝑝 𝜃 (x|z) + log 𝑝 𝜃 (z)] . z   = arg max log N (x; 𝑓 (Wz), diag(σx2 )) + log N (z; µz , diag(σz2 )) . z Each of the terms in this objective is a weighted squared error. For instance, the first term is the weighted squared error in reconstructing the data observation: log N (x; 𝑓 (Wz), diag(σx2 )) = 1 x − 𝑓 (Wz) 2 −𝑛x 1 log(2𝜋) − log |diag(σx2 )| − , 2 2 2 σx 2 where 𝑛x is the dimensionality of x and || · ||22 denotes the squared L2 norm. Plugging 131 µz z <latexit sha1_base64="MQNsW5zh2+4y3iJIQRBuQqhRWuw=">AAACjXicdVHLSgMxFE3HV62vVsGNm2ApuCozVdGFSNGFLiu0ttAZhkyaaYPJZEgyYh3nZ9zqD/k3Ztou+tALgcO5r5Nzg5hRpW37p2CtrW9sbhW3Szu7e/sH5crhsxKJxKSDBROyFyBFGI1IR1PNSC+WBPGAkW7wcp/nu69EKiqith7HxONoGNGQYqQN5ZeP3YCnrqJDjjLf5UiPgjB9z/xy1a7bk4CrwJmBKphFy68UfHcgcMJJpDFDSvUdO9ZeiqSmmJGs5CaKxAi/oCHpGxghTpSXTj6QwZphBjAU0rxIwwk735EirtSYB6Yyl6iWczn5V66f6PDaS2kUJ5pEeLooTBjUAuZuwAGVBGs2NgBhSY1WiEdIIqyNZwuT2o6X5uLyMQvrA56VanCeyWXEmr8t1iE2FGbDiP9DU7zaECuSGFfFwBhoTuIsH2AVPDfqznm98XRRbd7NjlMEJ+AUnAEHXIEmeAQt0AEYfIBP8AW+rQPr0rqxbqelVmHWcwQWwnr4BRl3y8A=</latexit> ÷ <latexit sha1_base64="RBARIkhKEuXaJ57u3HtEJ2V3Wh4=">AAACe3icdVHLSgMxFM2Mr1pfrS7dBIsgomWmKrosunFZoS9oh5LJpG1sHkOSEcvQf3Crf+bHCGbaLvrQC4HDuY9zcm8YM6qN53077sbm1vZObje/t39weFQoHje1TBQmDSyZVO0QacKoIA1DDSPtWBHEQ0Za4egpy7feiNJUiroZxyTgaCBon2JkLNXshjy9nvQKJa/sTQOuA38OSmAetV7R6XUjiRNOhMEMad3xvdgEKVKGYkYm+W6iSYzwCA1Ix0KBONFBOrU7geeWiWBfKvuEgVN2sSNFXOsxD20lR2aoV3MZ+Veuk5j+Q5BSESeGCDwT6icMGgmzv8OIKoING1uAsKLWK8RDpBA2dkNLk+p+kGbmsjFL8iGf5M/hIpPZiA1/X65DbCCtwpD/Q1O83hBrktitysgu0J7EXz3AOmhWyv5NufJyW6o+zo+TA6fgDFwAH9yDKngGNdAAGLyCD/AJvpwft+ReulezUteZ95yApXDvfgHKfMUP</latexit> <latexit sha1_base64="zU1t23bQbWEUhSwGFZ+dhUDLseE=">AAACeXicdVHLTgIxFO2ML8QX6NJNlZAQF2QGTHRJdOMSE14JTEin04HGdjppO0Qy4Rfc6q/5LW7sAAsG9CZNTs59nNN7/ZhRpR3n27L39g8OjwrHxZPTs/OLUvmyp0QiMeliwYQc+EgRRiPS1VQzMoglQdxnpO+/PWf5/oxIRUXU0fOYeBxNIhpSjHRGjQI6G5cqTt1ZBtwF7hpUwDra47I1HgUCJ5xEGjOk1NB1Yu2lSGqKGVkUR4kiMcJvaEKGBkaIE+WlS7MLWDVMAEMhzYs0XLKbHSniSs25byo50lO1ncvIv3LDRIePXkqjONEkwiuhMGFQC5j9HAZUEqzZ3ACEJTVeIZ4iibA2+8lN6rhempnLxuTkfb4oVuEmk9mINX/P1yE2EUZhyv+hKd5tiBVJzFZFYBZoTuJuH2AX9Bp1t1lvvN5XWk/r4xTANbgFNeCCB9ACL6ANugCDKfgAn+DL+rFv7Jp9tyq1rXXPFciF3fwFDZDESg==</latexit> <latexit sha1_base64="CEOswlEcYaImwi9o9uxLj0763T8=">AAACjXicdVFbS8MwGM3qbc7bVPDFl+AQfBqtF/RBZOiDPk7YVFhLSdN0C0uakqSyUftnfNU/5L8x3fqwi34QOJzvdnK+IGFUadv+qVgrq2vrG9XN2tb2zu5eff/gRYlUYtLFggn5FiBFGI1JV1PNyFsiCeIBI6/B8KHIv74TqaiIO3qcEI+jfkwjipE2lF8/cgOeuYr2Ocp9lyM9CKJslPv1ht20JwGXgVOCBiij7e9XfDcUOOUk1pghpXqOnWgvQ1JTzEhec1NFEoSHqE96BsaIE+Vlkw/k8NQwIYyENC/WcMLOdmSIKzXmgaksJKrFXEH+leulOrrxMhonqSYxni6KUga1gIUbMKSSYM3GBiAsqdEK8QBJhLXxbG5Sx/GyQlwxZm59wPPaKZxlChmJ5qP5OsT6wmwY8H9oipcbEkVS46oIjYHmJM7iAZbBy3nTuWieP182WvflcargGJyAM+CAa9ACT6ANugCDD/AJvsC3tWddWbfW3bTUqpQ9h2AurMdfFVHLvg==</latexit> ⇠z W W | <latexit sha1_base64="9fxCak7b507jN/DyEhOqRdmZkqA=">AAACi3icdVFbSwJBGB23m1mWVm+9DInQk+xqUEQPUgQ9GngDXWR2nNXBmZ1lZjbSZf9Lr/WP+jfNqoGX+uCDw/luh/N5IaNK2/Z3xtrZ3ds/yB7mjo7zJ6eF4llbiUhi0sKCCdn1kCKMBqSlqWakG0qCuMdIx5s8pfXOG5GKiqCppyFxORoF1KcYaUMNChd9jvTY8+NO8otmyaBQsiv2POA2cJagBJbRGBQzg/5Q4IiTQGOGlOo5dqjdGElNMSNJrh8pEiI8QSPSMzBAnCg3nstPYNkwQ+gLaTLQcM6uTsSIKzXlnulMJarNWkr+VetF2r9zYxqEkSYBXhzyIwa1gKkXcEglwZpNDUBYUqMV4jGSCGvj2NqmpuPGqbh0zdp5jye5MlxlUhmh5u/rfYiNhLkw5v/QFG8PhIpExlUxNAaalzibD9gG7WrFqVWqrzel+uPyOVlwCa7ANXDALaiDF9AALYDBDHyAT/Bl5a2adW89LFqtzHLmHKyF9fwDCinK2g==</latexit> ÷ <latexit sha1_base64="zU1t23bQbWEUhSwGFZ+dhUDLseE=">AAACeXicdVHLTgIxFO2ML8QX6NJNlZAQF2QGTHRJdOMSE14JTEin04HGdjppO0Qy4Rfc6q/5LW7sAAsG9CZNTs59nNN7/ZhRpR3n27L39g8OjwrHxZPTs/OLUvmyp0QiMeliwYQc+EgRRiPS1VQzMoglQdxnpO+/PWf5/oxIRUXU0fOYeBxNIhpSjHRGjQI6G5cqTt1ZBtwF7hpUwDra47I1HgUCJ5xEGjOk1NB1Yu2lSGqKGVkUR4kiMcJvaEKGBkaIE+WlS7MLWDVMAEMhzYs0XLKbHSniSs25byo50lO1ncvIv3LDRIePXkqjONEkwiuhMGFQC5j9HAZUEqzZ3ACEJTVeIZ4iibA2+8lN6rhempnLxuTkfb4oVuEmk9mINX/P1yE2EUZhyv+hKd5tiBVJzFZFYBZoTuJuH2AX9Bp1t1lvvN5XWk/r4xTANbgFNeCCB9ACL6ANugCDKfgAn+DL+rFv7Jp9tyq1rXXPFciF3fwFDZDESg==</latexit> <latexit sha1_base64="KEbVWs/M/y0aOeDnzmNP9sgSxLY=">AAACf3icdVHLTgIxFO2ML8QX6NJNIyG6IjNoou6IblxiwivChHRKBxr6mLQdI5nwF271v/wbO8CCAb1Jk5NzH+f03jBmVBvP+3Hcnd29/YPCYfHo+OT0rFQ+72iZKEzaWDKpeiHShFFB2oYaRnqxIoiHjHTD6XOW774TpakULTOLScDRWNCIYmQs9TbgyEzCKO3Oh6WKV/MWAbeBvwIVsIrmsOwMByOJE06EwQxp3fe92AQpUoZiRubFQaJJjPAUjUnfQoE40UG6sDyHVcuMYCSVfcLABbvekSKu9YyHtjKzqDdzGflXrp+Y6CFIqYgTQwReCkUJg0bC7P9wRBXBhs0sQFhR6xXiCVIIG7ul3KSWH6SZuWxMTj7k82IVrjOZjdjwj3wdYmNpFSb8H5ri7YZYk8RuVY7sAu1J/M0DbINOvebf1uqvd5XG0+o4BXAJrsAN8ME9aIAX0ARtgIEAn+ALfLuOe+3WXG9Z6jqrnguQC/fxFwCpxgU=</latexit> <latexit sha1_base64="RD+2LJCJnMHv1+m0E+1FMgx45sE=">AAACinicdVHLSsNAFJ3GV61VW7t0M1gKrkpSBRU3oi5cVugLmhAm00k7dCYTZiZiDfkWt/pJ/o2Ttos+9MKFw7mvw7lBzKjStv1TsHZ29/YPioelo/LxyWmletZTIpGYdLFgQg4CpAijEelqqhkZxJIgHjDSD6ZPeb3/RqSiIuroWUw8jsYRDSlG2lB+peYGPHXfaea7HOlJEKYfmV+p2017HnAbOEtQB8to+9WC744ETjiJNGZIqaFjx9pLkdQUM5KV3ESRGOEpGpOhgRHiRHnpXH0GG4YZwVBIk5GGc3Z1IkVcqRkPTGcuUW3WcvKv2jDR4a2X0ihONInw4lCYMKgFzK2AIyoJ1mxmAMKSGq0QT5BEWBvD1jZ1HC/NxeVr1s4HPCs14CqTy4g1f1/vQ2wszIUJ/4emeHsgViQxroqRMdC8xNl8wDbotZrOVbP1el1/eFw+pwjOwQW4BA64AQ/gBbRBF2AwA5/gC3xbZatl3Vn3i1arsJypgbWwnn8BGrXKcg==</latexit> Wz x <latexit sha1_base64="x2X+DDb3YI9YVUcVOCJwaRhTENQ=">AAACinicdVHLSsNAFJ3GV61VW7t0M1gKrkpSBRU3RV24rNAXNCFMppN2cCYTZiZiDfkWt/pJ/o2Ttos+9MKFw7mvw7lBzKjStv1TsHZ29/YPioelo/LxyWmletZXIpGY9LBgQg4DpAijEelpqhkZxpIgHjAyCF4f8/rgjUhFRdTVs5h4HE0iGlKMtKH8Ss0NeOryJPNdjvQ0CNOPzK/U7aY9D7gNnCWog2V0/GrBd8cCJ5xEGjOk1MixY+2lSGqKGclKbqJIjPArmpCRgRHiRHnpXH0GG4YZw1BIk5GGc3Z1IkVcqRkPTGcuUW3WcvKv2ijR4a2X0ihONInw4lCYMKgFzK2AYyoJ1mxmAMKSGq0QT5FEWBvD1jZ1HS/NxeVr1s4HPCs14CqTy4g1f1/vQ2wizIUp/4emeHsgViQxroqxMdC8xNl8wDbot5rOVbP1cl1vPyyfUwTn4AJcAgfcgDZ4Bh3QAxjMwCf4At9W2WpZd9b9otUqLGdqYC2sp18cyMpz</latexit> <latexit sha1_base64="RBARIkhKEuXaJ57u3HtEJ2V3Wh4=">AAACe3icdVHLSgMxFM2Mr1pfrS7dBIsgomWmKrosunFZoS9oh5LJpG1sHkOSEcvQf3Crf+bHCGbaLvrQC4HDuY9zcm8YM6qN53077sbm1vZObje/t39weFQoHje1TBQmDSyZVO0QacKoIA1DDSPtWBHEQ0Za4egpy7feiNJUiroZxyTgaCBon2JkLNXshjy9nvQKJa/sTQOuA38OSmAetV7R6XUjiRNOhMEMad3xvdgEKVKGYkYm+W6iSYzwCA1Ix0KBONFBOrU7geeWiWBfKvuEgVN2sSNFXOsxD20lR2aoV3MZ+Veuk5j+Q5BSESeGCDwT6icMGgmzv8OIKoING1uAsKLWK8RDpBA2dkNLk+p+kGbmsjFL8iGf5M/hIpPZiA1/X65DbCCtwpD/Q1O83hBrktitysgu0J7EXz3AOmhWyv5NufJyW6o+zo+TA6fgDFwAH9yDKngGNdAAGLyCD/AJvpwft+ReulezUteZ95yApXDvfgHKfMUP</latexit> ⇠x <latexit sha1_base64="e6aYWUJKyRseVz8YVR3NhNQYqj0=">AAACinicdVHLSsNAFJ3GV61VW7t0M1gKrkpSBRU3RV24rNAXNCFMppN2cCYTZibSEvItbvWT/BsnbRd96IULh3Nfh3ODmFGlbfunYO3tHxweFY9LJ+XTs/NK9aKvRCIx6WHBhBwGSBFGI9LTVDMyjCVBPGBkELw/5/XBB5GKiqir5zHxOJpENKQYaUP5lZob8NSd0cx3OdLTIExnmV+p2017EXAXOCtQB6vo+NWC744FTjiJNGZIqZFjx9pLkdQUM5KV3ESRGOF3NCEjAyPEifLShfoMNgwzhqGQJiMNF+z6RIq4UnMemM5cotqu5eRftVGiw3svpVGcaBLh5aEwYVALmFsBx1QSrNncAIQlNVohniKJsDaGbWzqOl6ai8vXbJwPeFZqwHUmlxFrPtvsQ2wizIUp/4emeHcgViQxroqxMdC8xNl+wC7ot5rOTbP1dltvP62eUwSX4ApcAwfcgTZ4BR3QAxjMwSf4At9W2WpZD9bjstUqrGZqYCOsl18Wj8pw</latexit> <latexit sha1_base64="ZkvLIu2TD+1jZIBra468ec3j8hg=">AAACi3icdVHJSgNBEO2MW4xbXG5eGoPgKcyooIgHUQSPEbJBMoaeTk3S2MvQ3SPGIf/iVf/Iv7En5mASLSh4vNoer6KEM2N9/6vgLS2vrK4V10sbm1vbO+XdvaZRqabQoIor3Y6IAc4kNCyzHNqJBiIiDq3o+S6vt15AG6Zk3Y4SCAUZSBYzSqyjeuWDriB2GMVZa/zUZdKCpoT3yhW/6k8CL4JgCipoGrXebqHX7SuaCpCWcmJMJ/ATG2ZEW0Y5jEvd1EBC6DMZQMdBSQSYMJvIH+Njx/RxrLRLafGE/T2REWHMSESuMxdr5ms5+Vetk9r4MsyYTFILkv4cilOOrcK5F7jPNFDLRw4QqpnTiumQaEKdDbOb6kGY5eLyNTPnIzEuHePfTC4jseJ1to/wgXIXhuIfmtHFgcRA6lxVfWege0kw/4BF0DytBmfV08fzys3t9DlFdIiO0AkK0AW6QQ+ohhqIojf0jj7Qp7flnXlX3vVPq1eYzuyjmfDuvwGeM8qm</latexit> z <latexit sha1_base64="JJBRHMAw1+FJL17Y9ikHFHSRDGA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuyIyaqDuiG5eY8IowIZ1OBxr6mLQdI074C7f6X/6NHWDBgN6kycm5j3N6bxAzqo3r/hScre2d3b3ifung8Oj4pFw57WiZKEzaWDKpegHShFFB2oYaRnqxIogHjHSDyVOW774RpakULTONic/RSNCIYmQs9TrgyIyDKP2YDctVt+7OA24CbwmqYBnNYaUwHIQSJ5wIgxnSuu+5sfFTpAzFjMxKg0STGOEJGpG+hQJxov10bnkGa5YJYSSVfcLAObvakSKu9ZQHtjKzqNdzGflXrp+Y6N5PqYgTQwReCEUJg0bC7P8wpIpgw6YWIKyo9QrxGCmEjd1SblLL89PMXDYmJx/wWakGV5nMRmz4e74OsZG0CmP+D03xZkOsSWK3KkO7QHsSb/0Am6BzXfdu6tcvt9XG4/I4RXAOLsAV8MAdaIBn0ARtgIEAn+ALfDsF59KpO+6i1Ckse85ALpyHX0lCxig=</latexit> <latexit sha1_base64="f85W8qlbhIMOhjGJ64NRDqqSQgA=">AAACf3icdVHLTgIxFC3jC/EFunTTSIiuJjNqou6IblxiwivChHQ6HWjoY9J2DGTCX7jV//Jv7AALHnqTJifnPs7pvWHCqDae91Nwdnb39g+Kh6Wj45PTs3LlvK1lqjBpYcmk6oZIE0YFaRlqGOkmiiAeMtIJxy95vvNBlKZSNM00IQFHQ0FjipGx1HufIzMK42wyG5SrnuvNA24DfwmqYBmNQaUw6EcSp5wIgxnSuud7iQkypAzFjMxK/VSTBOExGpKehQJxooNsbnkGa5aJYCyVfcLAObvakSGu9ZSHtjK3qDdzOflXrpea+DHIqEhSQwReCMUpg0bC/P8woopgw6YWIKyo9QrxCCmEjd3S2qSmH2S5uXzMmnzIZ6UaXGVyG4nhk/U6xIbSKoz4PzTF2w2JJqndqozsAu1J/M0DbIP2revfubdv99X68/I4RXAJrsAN8MEDqINX0AAtgIEAn+ALfDsF59pxHW9R6hSWPRdgLZynX0UcxiY=</latexit> x Figure 8.3: Hierarchical Predictive Coding. The diagram shows the basic computation graph for a Gaussian latent variable 36 model with MAP inference. The insets show the weighted error calculation for the latent (left) and observed (right) variables. these terms into the objective and dropping terms that do not depend on z yields # " −1 x − 𝑓 (Wz) 2 1 z − µz 2 ∗ , (8.4) − z = arg max z 2 σx 2 σz 2 2 = arg max ℒ(z; 𝜃), z where we have defined the objective as ℒ(z; 𝜃). For purposes of illustration, let us assume that 𝑓 (·) is the identity function, i.e. 𝑓 (Wz) = Wz. We can then evaluate the gradient of ℒ(z; 𝜃) w.r.t. z, yielding   z − µz | x − Wz ∇z ℒ(z; 𝜃) = W − . σx σz The transposed weight matrix, W| , comes from differentiating Wz, and translates the error in reconstruction into an update in z. If we define the following terms as weighted errors: ξx ≡ x − Wz , σx ξz ≡ z − µz , σz then we can re-write the gradient using these terms: ∇z ℒ(z; 𝜃) = W| ξx − ξz . (8.5) 132 Thus, if we want to perform inference using gradient-based optimization, e.g. z ← z + 𝛼∇z ℒ(z; 𝜃), we need 1) the weighted errors, ξx and ξz , and 2) the transposed weights, W| , or more generally, the Jacobian of the conditional likelihood mean. This overall scheme is depicted in Figure 8.3, using the computational graph format from previous chapters. To learn the weight parameters, we can differentiate ℒ(z; 𝜃) (Eq. 8.4) w.r.t. W:   x − Wz | z ∇W ℒ(z; 𝜃) = σx = ξx z | . This gradient is the product of a local error term, ξx , and the latent variable, z, possibly suggesting a biologically-plausible learning rule (Whittington and Bogacz, 2017). Predictive coding identifies the conditional likelihood (Eq. 8.2) with backward (or top-down) cortical projections, whereas inference updating (Eq. 8.5) is identified with forward (or bottom-up) cortical projections (K. Friston, 2005). Such connections are thought to be mediated by pyramidal neurons. Scaling this model up in size and structure, each cortical column could contain the necessary computational elements involved in predicting and estimating a latent variable. Interneurons within columns could be involved in error calculation (ξx and ξz ). Although we only discussed diagonal covariance matrices (σx2 and σz2 ), interneurons involved in local lateral inhibition could parameterize (the inverse of) full covariance matrices, i.e. 𝚺x and 𝚺z . This is an instance of spatial predictive coding, which we discussed in the previous section. Note that these factors, which weight ξx and ξz , effectively modulate the gain of each error term, possibly providing a form of “attention” (Feldman and K. Friston, 2010). Possible neural correspondences are summarized in Table 8.1. Here, we have discussed a simplified model of hierarchical predictive coding, with a single latent level and no dynamics. However, a full theory of hierarchical predictive coding would include these additional aspects and others. Karl Friston has explored various design choices throughout multiple papers (K. Friston et al., 2007; K. Friston, 2008a; K. Friston, 2008b), yet the core aspects of probabilistic generative modeling and variational inference remain largely the same. Further elaborating and comparing these design choices will be essential for empirically validating the details of hierarchical predictive coding. 133 Table 8.1: Proposed Neural Correspondences of Hierarchical Predictive Coding. Neuroscience Top-Down Cortical Projections Bottom-Up Cortical Projections Lateral Inhibition (Pyramidal) Neuron Activity Cortical Column Predictive Coding Generative Model Conditional Mapping Inference Updating Covariance Matrices Latent Variable Estimates & Errors Corresponding Estimate & Error Empirical Support Empirically validating predictive coding in neural circuits is an active area of research. This remains challenging, as it is difficult to disentangle the theory itself from the wide array of possible design choices, e.g. distributions, parameterizations, etc. (Gershman, 2019). Nevertheless, many of the core aspects of predictive coding do appear to have some empirical support. We briefly outline some of these studies here, but we refer the reader to the multiple review papers on the topic (Y. Huang and Rao, 2011; Bastos, Usrey, et al., 2012; Clark, 2013; Keller and Mrsic-Flogel, 2018; Walsh et al., 2020). Spatiotemporal As discussed above, various works have investigated spatiotemporal predictive coding in early sensory areas, primarily retina (Srinivasan, Laughlin, and Dubs, 1982; Atick and Redlich, 1992). This typically involves fitting retinal ganglion cell responses to a spatial whitening (or decorrelation) process (Graham, Chandler, and Field, 2006; Pitkow and Meister, 2012), which is dynamically adjusted based on lighting conditions (Hosoya, Baccus, and Meister, 2005). Similar analyses suggest that retina employs temporal predictive coding as well (Srinivasan, Laughlin, and Dubs, 1982; Palmer et al., 2015). While the exact mathematical details of these neural computations have not been fully characterized, the corresponding models contain stages of linear decorrelating filters (e.g. center-surround) followed by non-linearities. Importantly, non-linearities have been shown to be an essential aspect in explaining retinal ganglion cell responses (Pitkow and Meister, 2012), possibly inducing an added degree of sparsity (Graham, Chandler, and Field, 2006). As previously noted, similar spatiotemporal predictive coding computations may be found in thalamus (Dong and Atick, 1995) and cortex. While Dan, Atick, and Reid, 1996 provide some supporting evidence, such investigations are complicated by the presence of backward and modulatory interactions. 134 Hierarchical Early work toward empirically validating hierarchical predictive coding came from explaining extra-classical receptive field effects (Rao and Ballard, 1999; Rao and Sejnowski, 2002), whereby top-down processing in cortex can alter classical visual receptive fields, suggesting that top-down influences play an important role in sensory processing (Gilbert and Sigman, 2007). Likewise, temporal influences have been demonstrated in the form of repetition suppression (Summerfield et al., 2006), in which cortical activity diminishes in response to repeated, i.e. predictable, stimuli. This effect may reflect the suppression of errors through improved predictions. Predictive coding has also been postulated as an explanation of biphasic responses in LGN (Jehee and Ballard, 2009), in which reversing the visual input with an anti-correlated image results in a large neural response, presumably due to prediction errors. Predictive signals have been documented in auditory (Wacongne et al., 2011) and visual (T. Meyer and Olson, 2011) processing. Activity seemingly corresponding to prediction errors has also been observed in a variety of areas and contexts, including visual flow in primary visual cortex in mice (Keller, Bonhoeffer, and Hübener, 2012; Zmarz and Keller, 2016), auditory cortex in monkeys (Eliades and Wang, 2008) and rodents (Parras et al., 2017), and visual cortex in humans (S. O. Murray et al., 2002; Alink et al., 2010; Egner, Monti, and Summerfield, 2010). While further studies are needed, it appears that sensory cortex is engaged in some form of hierarchical and temporal prediction, with prediction error signals playing a key role in driving the perceptual process. The empirical evidence for spatiotemporal and hierarchical predictive coding is suggestive, but given the complexity of neural systems, some aspects of the theory are undoubtedly incorrect, incomplete, or under-specified. In particular, the complexity of biological systems makes it is difficult to isolate and assess detailed aspects of predictive coding. For instance, it appears that cortex calculates some form of prediction error, but without access to fine-grained recordings of all relevant signals, e.g. dendritic currents, neuromodulators, etc., it is difficult to determine the exact computational form of the circuit. Thus, while general aspects of predictive coding appear supported, we are unable to probe into the details of such models, making predictive coding a largely normative theory. One of the purposes of this chapter, and this thesis more broadly, has been to establish connections between predictive coding and machine learning. Ideally, by building larger-scale models and training them on similar sensory data, we can form more fine-grained empirical predictions for biological neural systems. Building off of the example of Rao and Ballard, 1999, Lotter, Kreiman, and Cox, 2018 provided another step in this direction, comparing 135 the responses of neural systems and their hierarchical predictive coding model. In the current thesis, we have attempted to help further build the foundation for this collaborative effort. 8.3 Connections We now discuss the connections between the ideas developed in this thesis and their inspirations from predictive coding. Iterative Amortization Iterative amortized inference (Chapter 3) was inspired by the inference scheme proposed by Rao and Ballard, 1999 and K. Friston, 2005. In these early formulations of hierarchical predictive coding, approximate inference is performed using gradientbased optimization of a point estimate of the latent variables. These works made it clear that prediction (or reconstruction) errors drive both inference and learning optimization, and this procedure can be readily extended to sequential settings (K. Friston, 2008b). However, such procedures typically assume that the inference gradients, supplied by forward connections, can be easily calculated, but the weights of these forward connections are, in fact, the Jacobian of the backward connections (Rao and Ballard, 1999) (Section 8.2). This is an example of the weight transport problem (Grossberg, 1987), i.e. the weights of one set of connections (forward) depends on the weights from another set of connections (backward). This is generally regarded as not being biologically-plausible. Amortization (Dayan et al., 1995) provides a simple solution to this problem: learn to perform inference optimization. That is, rather than transporting the generative weights to the inference connections, amortization learns a separate set of inference weights, potentially using similar local learning rules (Y. Bengio, 2014; Lee et al., 2015). Thus, despite criticism from K. Friston, 2018, amortization may offer a more biologically-plausible account of inference. Further, as demonstrated in Chapters 3, 4, and 6, by using non-linear functions, amortization is capable of automatically adjusting update step sizes, yielding accurate estimates with exceedingly few inference iterations. These substantial benefits in computational efficiency provide another argument for amortization over the gradient-based schemes often employed in predictive coding. Iterative amortization is an example of the more general approach of negative feedback. As noted at the beginning of this thesis, negative feedback was the core concept of cybernetics, which went on to inspire predictive coding. While hierarchical 136 predictive coding has largely focused on perceptual inference in cortex, the principles of negative feedback appear to apply more broadly to neural systems. Indeed, even at the outset of cybernetics, it was clear that cerebellum plays a central role in negative feedback control (Wiener, 1948). From more recent studies of cerebellum and other cerebellum-like structures (Ito, 1998; Bell, 2001; Kennedy et al., 2014), we are beginning to understand how such circuits correct sensorimotor prediction errors. One prominent example is given by the Purkinje cells of the cerebellum, which appear to take in error signals as inputs and output motor corrections. This follows the general paradigm of iterative amortization, mapping errors to updates. Casting these neural circuits in terms of amortization, i.e. learned negative feedback, may provide insights into how such error-correcting mechanisms are learned from experience. Sequential Autoregressive Flows The technique of sequential autoregressive flows (Chapter 5) was, in part, inspired by the temporal normalization schemes from Srinivasan, Laughlin, and Dubs, 1982 and Dong and Atick, 1995, which are thought to occur in retina and first-order relays of thalamus. Unlike these earlier works, which were limited to linear functions of previous inputs, sequential autoregressive flows can utilize non-linear functions to parameterize the normalizing affine transform. Likewise, because these feedforward transforms are learned using the prediction errors on the normalized variables, they can adapt to meet the demands or limitations of higher-level models (Figure 5.6a). A related technique is that of generalized coordinates (K. Friston, 2008a), decomposing a sequence into its temporal derivatives. Friston has suggested that this may be a general modeling technique employed by neural circuits. As we have seen, a simplified version of sequential autoregressive flows, using the previous variable as the affine shift, extracts an approximation of temporal derivatives (Section 5.2). Thus, given a short enough time step, Δ𝑡 = 𝑡1 − 𝑡0 , sequential autoregressive flows provide a technique for automatically learning an approximation of generalized coordinates. In Chapter 7, we saw that the same technique can be applied to control, serving as a low-level dynamical policy, i.e. a dynamical motor basis for control. While early sensory processing and motor processing are often considered separately, we see that spatiotemporal dependencies are central to both areas. Normalization (and its inverse) plays a singular role in both cases, simplifying estimation for upstream models. If normalization is truly a canonical neural computation (Carandini and Heeger, 137 2012), then similar temporal normalization operations may parameterize dynamics estimation throughout cortex, operating in conjunction with spatial normalization. We saw this in Chapter 5, where the modified VideoFlow model (Kumar et al., 2020) utilized spatial normalization within time steps and temporal normalization across time steps, demonstrating the utility of these complementary procedures. We have exclusively investigated normalizing flows with affine transforms (Dinh, Sohl-Dickstein, and S. Bengio, 2017), due to their simplicity and their similarity to proposed neural computations. Given their connection with Gaussian densities and Gaussianization (Chen and Gopinath, 2001) (normalization is, ultimately, the process of converting a data distribution into a standard Normal (Gaussian) density), transforms of this form are prominent in the statistics literature (Friedman, 1987; Kessy, Lewin, and Strimmer, 2018). However, the change of variables formula readily applies to all invertible transforms, including non-affine transforms. This more general perspective, afforded by normalizing flows, offers a method toward improving spatiotemporal normalization in current models. Similarly, it suggests that neural circuits may implement multiple computational forms of normalization transforms, ranging from simple, affine or constant transforms to more complex, non-affine transforms (Durkan et al., 2019). These may rely on computational mechanisms within individual neurons as well as spatiotemporal interactions between neurons. 8.4 Correspondences Having drawn connections between the machine learning approaches developed in this thesis and their inspirations in predictive coding, we can now traverse this bridge from machine learning, through predictive coding, to neuroscience. In this section, we identify correspondences implied by this bridge. In particular, we explore the consequences of two implied correspondences: 1) pyramidal neurons and deep networks and 2) lateral inhibition and normalizing flows. These correspondences should be interpreted at a functional level, potentially shedding new light on the computational and learning mechanisms employed in biological neural circuits. In certain aspects, these correspondences offer a substantial departure from the current paradigm linking biological and artificial neural networks, providing an alternative approach toward connecting these areas. Hierarchical predictive coding and deep latent variable models, particularly variational autoencoders (VAEs) (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014), are highly related in both their model formulations and inference approaches 138 J| ✓ ✓ ✓ <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="Gch/JFSva5F3XM1dLnxjDscUgyE=">AAACi3icdVHJSgNBEO2MW4xbXG5eGoPgKcyooIgHUQTxpJCYQDKGnk5N0tjL0N0jxiH/4lX/yL+xJ8nBJFpQ8Hi1PV5FCWfG+v53wVtYXFpeKa6W1tY3NrfK2ztPRqWaQp0qrnQzIgY4k1C3zHJoJhqIiDg0opebvN54BW2YkjU7SCAUpCdZzCixjuqU99qC2H4UZ/fD5zaTFjQlvFOu+FV/FHgeBBNQQZN46GwXOu2uoqkAaSknxrQCP7FhRrRllMOw1E4NJIS+kB60HJREgAmzkfwhPnRMF8dKu5QWj9jfExkRxgxE5DpzsWa2lpN/1Vqpjc/DjMkktSDp+FCccmwVzr3AXaaBWj5wgFDNnFZM+0QT6myY3lQLwiwXl6+ZOh+JYekQ/2ZyGYkVb9N9hPeUu9AX/9CMzg8kBlLnquo6A91LgtkHzIOn42pwUj1+PK1cXU+eU0T76AAdoQCdoSt0hx5QHVH0jj7QJ/ryNrwT78K7HLd6hcnMLpoK7/YHgrrKmQ==</latexit> (a) Predictive Coding <latexit sha1_base64="lxiql3JCVAJFF+6oZSBqr8bbMtc=">AAACeXicdVHLTgIxFC3jC/EFunRTJSTEBZkBE10S3bjEhFcCE9IphWlop03bMZIJv+BWf81vcWMHWDCgN2lycu7jnN4bSEa1cd3vnLO3f3B4lD8unJyenV8US5ddLWKFSQcLJlQ/QJowGpGOoYaRvlQE8YCRXjB7TvO9N6I0FVHbzCXxOZpGdEIxMik1lCEdFctuzV0G3AXeGpTBOlqjUm40HAsccxIZzJDWA8+Vxk+QMhQzsigMY00kwjM0JQMLI8SJ9pOl2QWsWGYMJ0LZFxm4ZDc7EsS1nvPAVnJkQr2dS8m/coPYTB79hEYyNiTCK6FJzKARMP05HFNFsGFzCxBW1HqFOEQKYWP3k5nU9vwkNZeOycgHfFGowE0mtSENf8/WITYVViHk/9AU7zZITWK7VTG2C7Qn8bYPsAu69ZrXqNVf78vNp/Vx8uAa3IIq8MADaIIX0AIdgEEIPsAn+Mr9ODdO1blblTq5dc8VyITT+AUJg8RI</latexit> <latexit sha1_base64="lxiql3JCVAJFF+6oZSBqr8bbMtc=">AAACeXicdVHLTgIxFC3jC/EFunRTJSTEBZkBE10S3bjEhFcCE9IphWlop03bMZIJv+BWf81vcWMHWDCgN2lycu7jnN4bSEa1cd3vnLO3f3B4lD8unJyenV8US5ddLWKFSQcLJlQ/QJowGpGOoYaRvlQE8YCRXjB7TvO9N6I0FVHbzCXxOZpGdEIxMik1lCEdFctuzV0G3AXeGpTBOlqjUm40HAsccxIZzJDWA8+Vxk+QMhQzsigMY00kwjM0JQMLI8SJ9pOl2QWsWGYMJ0LZFxm4ZDc7EsS1nvPAVnJkQr2dS8m/coPYTB79hEYyNiTCK6FJzKARMP05HFNFsGFzCxBW1HqFOEQKYWP3k5nU9vwkNZeOycgHfFGowE0mtSENf8/WITYVViHk/9AU7zZITWK7VTG2C7Qn8bYPsAu69ZrXqNVf78vNp/Vx8uAa3IIq8MADaIIX0AIdgEEIPsAn+Mr9ODdO1blblTq5dc8VyITT+AUJg8RI</latexit> (b) VAE (Direct) (c) VAE (Iterative) Figure 8.4: Hierarchical Predictive Coding & VAEs. Computation39 diagrams for 37 38 (a) hierarchical predictive coding, (b) VAE with direct amortized inference, and (c) VAE with iterative amortized inference (Chapter 3). J| denotes the transposed Jacobian matrix of the generative model’s conditional likelihood. Red dotted lines denote gradients, and black dashed lines denote amortized inference. Hierarchical predictive coding and VAEs are highly similar in both their model formulation and inference approach. (Figure 8.4). Specifically, • Model Formulation: Both areas focus on hierarchical latent Gaussian models with non-linear dependencies between latent levels, as well as dependencies within levels via covariance matrices (predictive coding) or normalizing flows (VAEs). Note that a covariance matrix is computationally equivalent to an affine normalizing flow with linear dependencies (Kingma, Salimans, et al., 2016). • Inference: Both areas use variational inference, often with Gaussian approximate posteriors. While predictive coding employs gradient-based optimization and VAEs employ amortized optimization, these are just different design choices in solving the same inference optimization problem. While previous works within predictive coding and deep latent variable models have explored distinct design choices, e.g. in parameterizing dynamics, the two areas share a common mathematical foundation, inherited from cybernetics and descendant areas. With this connection explicitly established, we can now identify the biological correspondences implied by the connection from VAEs to predictive coding to neuroscience. Looking at Table 8.1, we see that top-down and bottom-up cortical 139 projections, each mediated by pyramidal neurons, respectively parameterize the generative model and inference updates. Mapping this onto VAEs implies that deep (artificial) neural networks are in correspondence with pyramidal neuron dendrites (Figure 8.5). This analogy is not perfect, as each output dimension of a deep network shares parameters with the other outputs through previous layers. In this regard, the analogy to pyramidal dendrites in cortex would specifically imply a separate deep network per variable in a VAE. Or, conversely, a deep network corresponds to a collection of pyramidal dendrites operating in parallel. Lateral inhibitory interneurons, which parameterize the inverse covariance matrices at each latent level in predictive coding, map onto normalizing flows. As mentioned above, normalizing flows are a non-linear generalization of linear covariance matrices, suggesting the possibility of non-linear normalization computations in cortex and elsewhere. These correspondences are obviously quite coarse-grained, and many details are left to be filled-in. However, they may provide a useful starting point for shifting the current analogies between machine learning and neuroscience. Below, we explore some of the consequences of these correspondences. Pyramidal Neurons & Deep Networks Non-linear Dendritic Computation Placing deep networks in correspondence with pyramidal dendrites departs from the traditional one-to-one correspondence of biological and artificial neurons (McCulloch and Pitts, 1943). This suggests that (some) individual biological neurons may be better computationally described as non-linear functions. Evidence from neuroscience supports this analogy. Early work in simulations proposed that individual pyramidal neurons, through dendritic processing, could operate as multi-layer artificial networks (Zador, Claiborne, and Brown, 1992; Mel, 1992). This was later supported by empirical findings that hippocampal pyramidal dendrites act as computational ‘subunits,’ yielding the equivalent of a two-layer artificial network (Poirazi, Brannon, and Mel, 2003; Polsky, Mel, and Schiller, 2004). More recently, Gidon et al., 2020 demonstrated that individual L2/3 pyramidal neurons are capable of computing the XOR operation, known for requiring non-linear processing (Minsky and Papert, 1969). This potential correspondence between deep networks and pyramidal dendrites posits a substantial role for dendritic computation (London and Häusser, 2005), moving beyond the overly simplistic comparison of biological and artificial neurons. Further, rather than assuming every neuron is equivalent, i.e. linear summation with non-linearity, separate classes of neurons would represent distinct function classes, likely derived 140 pyramidal neuron <latexit sha1_base64="7XY8gHJwTUsdq0+DWdSkGUzfTfM=">AAACkHicdVFdSxtBFJ1sa2tjbaN9Kn0ZDEKfwq4W6pvWvpQ+WTAqJEu4O3uTDM4XM3fFsIT+Gl/b3+O/cTbmwZj2wsDh3I9z59zCKRkoTe9byYuXG69eb75pb73dfve+s7N7EWzlBfaFVdZfFRBQSYN9kqTwynkEXSi8LK6/N/nLG/RBWnNOM4e5homRYymAIjXqfBwS3pLXtZt50LIExQ1W3pr5qNNNe+ki+DrIlqDLlnE22mmNhqUVlUZDQkEIgyx1lNfgSQqF8/awCuhAXMMEBxEa0BjyevGHOd+PTMnH1sdniC/Ypx016BBmuoiVGmganuca8l+5QUXjo7yWxlWERjwKjSvFyfLGEF5Kj4LULAIQXsZduZiCB0HRtpVJ51leN8s1Y1bkCz1v7/OnTLOGI327WgdqYqPCVP+HlmK9wQWsoqu2jAbGk2TPD7AOLg562WHv4NeX7snp8jib7BPbY59Zxr6yE/aDnbE+E+w3u2N/2N9kNzlKjpNvj6VJa9nzga1E8vMBR4zNIQ==</latexit> J| ✓ <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="Gch/JFSva5F3XM1dLnxjDscUgyE=">AAACi3icdVHJSgNBEO2MW4xbXG5eGoPgKcyooIgHUQTxpJCYQDKGnk5N0tjL0N0jxiH/4lX/yL+xJ8nBJFpQ8Hi1PV5FCWfG+v53wVtYXFpeKa6W1tY3NrfK2ztPRqWaQp0qrnQzIgY4k1C3zHJoJhqIiDg0opebvN54BW2YkjU7SCAUpCdZzCixjuqU99qC2H4UZ/fD5zaTFjQlvFOu+FV/FHgeBBNQQZN46GwXOu2uoqkAaSknxrQCP7FhRrRllMOw1E4NJIS+kB60HJREgAmzkfwhPnRMF8dKu5QWj9jfExkRxgxE5DpzsWa2lpN/1Vqpjc/DjMkktSDp+FCccmwVzr3AXaaBWj5wgFDNnFZM+0QT6myY3lQLwiwXl6+ZOh+JYekQ/2ZyGYkVb9N9hPeUu9AX/9CMzg8kBlLnquo6A91LgtkHzIOn42pwUj1+PK1cXU+eU0T76AAdoQCdoSt0hx5QHVH0jj7QJ/ryNrwT78K7HLd6hcnMLpoK7/YHgrrKmQ==</latexit> deep network <latexit sha1_base64="J6hfAr7RJIeDOwMd/BEx4vgzo8Y=">AAACjHicdVFdSxtBFJ2stcbUarT41JehQehT2FVBQQrBgvTRglEhWcLs7E0yZL6YuWsNS36Mr/qL+m+cjXkwRi9cOJz7dTg3s1J4jOP/tWjt0/rnjfpm48vW1+2d5u7etTeF49DlRhp3mzEPUmjookAJt9YBU5mEm2zyu6rf3IHzwugrnFpIFRtpMRScYaAGzf0+wj06VeYAlmrAf8ZNZoNmK27H86CrIFmAFlnE5WC3NujnhhcKNHLJvO8lscW0ZA4FlzBr9AsPlvEJG0EvQM0U+LSc65/Rg8DkdGhcSI10zr6eKJnyfqqy0KkYjv3bWkW+V+sVODxNS6FtgaD5y6FhISkaWplBc+GAo5wGwLgTQSvlY+YYx2DZ0qarJC0rcdWapfOZmjUO6GumkmFR3S/3MTky4cJYfUALvjpgPRTBVZMHA8NLkrcPWAXXh+3kqH3497jVOV88p06+kx/kJ0nICemQP+SSdAknJXkgj+Qp2o6Oo7Po10trVFvMfCNLEV08AwaCy0c=</latexit> 42 Figure 8.5: Pyramidal Neurons & Deep Networks. Connecting deep latent variable models with predictive coding places deep networks (bottom) in correspondence with the dendrites of pyramidal neurons (top). This is in contrast with conventional oneto-one analogies of biological and artificial neurons, suggesting a larger role for nonlinear dendritic computation and alternative correspondences for backpropagation. from their morphology. This places a greater emphasis on understanding neural circuits instead of assuming a uniform network of identical computational elements. Amortization It is feasible that similar hardware and mechanisms underlying generative predictions could also perform inference updating, i.e. amortization. This is the insight of deep latent variable models: deep networks can parameterize conditional probabilities in both directions. The computational components are identical, with different inputs and output targets. Building off of the correspondence of pyramidal dendrites and deep networks, in cortex, we see particular classes of pyramidal neurons with separate apical and basal dendrites. These segregated dendritic compartments selectively take inputs from top-down and bottom-up pathways respectively (Bekkers, 2011; Guergiuev, Lillicrap, and Richards, 2016; Richards, 2019), thought to perform separate computations. These pyramidal neurons could implement a form of iterative amortized inference model (Chapter 3), separately processing top-down and bottomup error signals to update inference estimates (Figure 8.6). This agrees with the conjecture from predictive coding that separate neurons in the forward pathway perform inference updating. Amortization also resolves the weight-transport issue from predictive coding, as separate inference weights are learned. While some empirical evidence appears to support amortization (Yildirim et al., 2015; Dasgupta et 141 pyramidal neuron <latexit sha1_base64="7XY8gHJwTUsdq0+DWdSkGUzfTfM=">AAACkHicdVFdSxtBFJ1sa2tjbaN9Kn0ZDEKfwq4W6pvWvpQ+WTAqJEu4O3uTDM4XM3fFsIT+Gl/b3+O/cTbmwZj2wsDh3I9z59zCKRkoTe9byYuXG69eb75pb73dfve+s7N7EWzlBfaFVdZfFRBQSYN9kqTwynkEXSi8LK6/N/nLG/RBWnNOM4e5homRYymAIjXqfBwS3pLXtZt50LIExQ1W3pr5qNNNe+ki+DrIlqDLlnE22mmNhqUVlUZDQkEIgyx1lNfgSQqF8/awCuhAXMMEBxEa0BjyevGHOd+PTMnH1sdniC/Ypx016BBmuoiVGmganuca8l+5QUXjo7yWxlWERjwKjSvFyfLGEF5Kj4LULAIQXsZduZiCB0HRtpVJ51leN8s1Y1bkCz1v7/OnTLOGI327WgdqYqPCVP+HlmK9wQWsoqu2jAbGk2TPD7AOLg562WHv4NeX7snp8jib7BPbY59Zxr6yE/aDnbE+E+w3u2N/2N9kNzlKjpNvj6VJa9nzga1E8vMBR4zNIQ==</latexit> J| ✓ <latexit sha1_base64="y9Nz2T+fmSMi/3DG5f53pxwGB6k=">AAACe3icdVHLSgMxFE3HV61vXboJFkFEyowPdCm6cVnBPqAdSia9baPJZEjuiGXoP7jVP/NjBDO1iz70QuBw7uOc3BslUlj0/a+Ct7S8srpWXC9tbG5t7+zu7detTg2HGtdSm2bELEgRQw0FSmgmBpiKJDSil/s833gFY4WOn3CYQKhYPxY9wRk6qt7GASDr7Jb9ij8OugiCCSiTSVQ7e4VOu6t5qiBGLpm1rcBPMMyYQcEljErt1ELC+AvrQ8vBmCmwYTa2O6LHjunSnjbuxUjH7HRHxpS1QxW5SsVwYOdzOflXrpVi7ybMRJykCDH/FeqlkqKm+d9pVxjgKIcOMG6E80r5gBnG0W1oZtJTEGa5uXzMjHykRqVjOs3kNhJUb7N1TPa1Uxiof2jBFxsSC6nbqu66BbqTBPMHWAT180pwUTl/vCzf3k2OUySH5IickIBck1vyQKqkRjh5Ju/kg3wWvr2yd+qd/ZZ6hUnPAZkJ7+oHEUPFMQ==</latexit> <latexit sha1_base64="Gch/JFSva5F3XM1dLnxjDscUgyE=">AAACi3icdVHJSgNBEO2MW4xbXG5eGoPgKcyooIgHUQTxpJCYQDKGnk5N0tjL0N0jxiH/4lX/yL+xJ8nBJFpQ8Hi1PV5FCWfG+v53wVtYXFpeKa6W1tY3NrfK2ztPRqWaQp0qrnQzIgY4k1C3zHJoJhqIiDg0opebvN54BW2YkjU7SCAUpCdZzCixjuqU99qC2H4UZ/fD5zaTFjQlvFOu+FV/FHgeBBNQQZN46GwXOu2uoqkAaSknxrQCP7FhRrRllMOw1E4NJIS+kB60HJREgAmzkfwhPnRMF8dKu5QWj9jfExkRxgxE5DpzsWa2lpN/1Vqpjc/DjMkktSDp+FCccmwVzr3AXaaBWj5wgFDNnFZM+0QT6myY3lQLwiwXl6+ZOh+JYekQ/2ZyGYkVb9N9hPeUu9AX/9CMzg8kBlLnquo6A91LgtkHzIOn42pwUj1+PK1cXU+eU0T76AAdoQCdoSt0hx5QHVH0jj7QJ/ryNrwT78K7HLd6hcnMLpoK7/YHgrrKmQ==</latexit> deep network <latexit sha1_base64="J6hfAr7RJIeDOwMd/BEx4vgzo8Y=">AAACjHicdVFdSxtBFJ2stcbUarT41JehQehT2FVBQQrBgvTRglEhWcLs7E0yZL6YuWsNS36Mr/qL+m+cjXkwRi9cOJz7dTg3s1J4jOP/tWjt0/rnjfpm48vW1+2d5u7etTeF49DlRhp3mzEPUmjookAJt9YBU5mEm2zyu6rf3IHzwugrnFpIFRtpMRScYaAGzf0+wj06VeYAlmrAf8ZNZoNmK27H86CrIFmAFlnE5WC3NujnhhcKNHLJvO8lscW0ZA4FlzBr9AsPlvEJG0EvQM0U+LSc65/Rg8DkdGhcSI10zr6eKJnyfqqy0KkYjv3bWkW+V+sVODxNS6FtgaD5y6FhISkaWplBc+GAo5wGwLgTQSvlY+YYx2DZ0qarJC0rcdWapfOZmjUO6GumkmFR3S/3MTky4cJYfUALvjpgPRTBVZMHA8NLkrcPWAXXh+3kqH3497jVOV88p06+kx/kJ0nICemQP+SSdAknJXkgj+Qp2o6Oo7Po10trVFvMfCNLEV08AwaCy0c=</latexit> 43 Figure 8.6: Pyramidal Neurons & Amortization. In predictive coding, inference updating is implemented using forward pyramidal neurons in cortex, taking prediction errors as input. In deep latent variable models, iterative amortized inference plays a similar role, continuing the analogy of pyramidal neurons and deep networks. Interestingly, this suggests a separation of processing in apical and basal dendrites, incorporating errors from the current and lower latent level. al., 2018), we note that K. Friston, 2018 remains skeptical of its biological-plausibility. Further experiments, particularly at the cortical circuit level, are needed to resolve this question. For instance, amortization would require some form of stochastic gradient estimation, e.g. reparameterization gradients (Chapter 2). Although all of the necessary error signals are local to the amortized optimizer, the details remain unclear. Neural Oscillations Oscillations are a common feature of neural circuits, giving rise to various frequency bands in the local field potential (LFP). These frequencies are thought to arise from the synchronous activity of populations of neurons, such as recurrent activity in hippocampus resulting in the theta frequency band (4–10 Hz). Bastos, Vezoli, et al., 2015 identify distinct frequency bands associated with forward (gamma, 30–80 Hz) and backward (beta, 10–30 Hz) activity in neocortex. Likewise, Walsh et al., 2020, reviewing the literature, note that violations of expectations, i.e. large prediction errors, are associated with increased gamma amplitude, whereas beta amplitude tends to increase in preparation for a predicted stimulus (Fujioka et al., 2009). In agreement with hierarchical predictive coding, this supports the conjecture that backward projections convey predictions, while forward projections perform inference updating using prediction errors. If we consider the gamma and beta 142 Figure 8.7: Backpropagation. Placing deep networks in correspondence with pyramidal neuron dendrites suggests an alternative perspective on the biologicalplausibility of backpropagation. In deep latent variable models, backpropagation is only performed across variables that are44directly connected through a conditional probability (left). From the perspective presented here, this corresponds to learning within pyramidal neurons. One possible implementation may be through backpropagating action potentials, perhaps combined with other neuromodulatory inputs (right). frequency bands as distinct network-wide “clock rates” associated with inference and dynamics estimation, respectively, this suggests that cortex performs anywhere from 1–8 inference iterations per time step. Each time step would constitute 30–100 ms. As seen with amortized variational filtering in Chapter 4, as well iterative amortized policy optimization in Chapter 6, even with a single inference iteration per time step, amortization can yield reasonably accurate inference estimates. Note that direct amortization (Dayan et al., 1995) would suggest that these frequencies should be identical, in disagreement with empirical observations. Gradient-based optimization, in contrast, as suggested and employed by Rao and Ballard, 1999 and K. Friston, 2005, may not be sufficient to provide accurate inference estimates with so few iterations. This may provide further support for the claim that forward pyramidal neurons in cortex implement a form of iterative amortized inference. Backpropagation Training deep networks at scale appears to require gradientbased parameter optimization, i.e. backpropagation (Werbos, 1974; Rumelhart, Geoffrey E Hinton, and R. J. Williams, 1986). However, the biological plausibility of backpropagation remains an open question (Lillicrap, Santoro, et al., 2020). Critics argue that backpropagation requires non-local learning signals and other techniques (Grossberg, 1987; Crick, 1989), whereas the brain relies largely on local learning rules (Hebb, 1949; Markram et al., 1997; Bi and Poo, 1998). A number 143 of “biologically-plausible” formulations of backpropagation have been proposed (Stork, 1989; Körding and König, 2001; Xie and Seung, 2003; Geoffrey E. Hinton, 2007; Lillicrap, Cownden, et al., 2016), attempting to reconcile this disparity and others. Yet, consensus is still lacking regarding the biological implementations of these proposed techniques. From another perspective, the apparent biological implausibility of backpropagation may instead be the result of incorrectly assuming a one-to-one correspondence between biological and artificial neurons. If deep networks are in correspondence with pyramidal dendrites, this suggests a different perspective on the biological-plausibility of backpropagation. In hierarchical latent variable models, prediction errors at each level of the latent hierarchy provide a local learning signal (K. Friston, 2005; Y. Bengio, 2014; Lee et al., 2015; Whittington and Bogacz, 2017). Thus, the global objective is decomposed into local errors, with learning within each latent level performed through gradient-based optimization. This is exemplified by deep latent variable models, which utilize backpropagation within each latent level, but not (necessarily) across latent levels. Again, considering the correspondence of pyramidal dendrites and deep networks, this suggests that learning within pyramidal neurons may be more analogous to backpropagation (Figure 8.7). Not surprisingly, one possible candidate is backpropagating action potentials (G. J. Stuart and Sakmann, 1994; S. R. Williams and G. J. Stuart, 2000). These occur in the dendrites of pyramidal neurons, actively propagating a signal of neural activity back to synaptic inputs (G. Stuart et al., 1997; Brunner and Szabadics, 2016). This results in the location-dependent influx of calcium, leading to a variety of synaptic changes throughout the dendritic tree (Johenning et al., 2015). Indeed, Schiess, Urbanczik, and Senn, 2016 recently investigated a computational model of gradient backpropagation within dendritic trees. While many details remain unclear, this overall perspective of backpropagation within neurons, rather than across networks of neurons, offers a more biologically-plausible alternative; all signals are local to the pyramidal neurons/dendrites within the cortical circuit. Given the proposed theoretical role and empirical observations, this possible correspondence between backpropagation and backpropagating action potentials warrants further investigation. Lateral Inhibition & Normalizing Flows Sensory Input Normalization Sensory stimuli are highly redundant in both space and time. Examples include luminance at neighboring photoreceptors or pressure on adjacent mechanoreceptors, each of which tend to persist over time intervals. One 144 of the key computational roles of early sensory areas, e.g. retina, appears to be in reducing these redundancies through normalization. In retina, such normalization operations are carried out through lateral inhibition via horizontal and amacrine cells. As a result, the transmitted output signals are less correlated (Graham, Chandler, and Field, 2006; Pitkow and Meister, 2012). As we have discussed in this chapter and elsewhere, normalization and prediction are inseparable, i.e. one must form a prediction in order to normalize. Accordingly, previous works have framed early sensory processing in terms of (spatiotemporal) predictive coding (Srinivasan, Laughlin, and Dubs, 1982; Hosoya, Baccus, and Meister, 2005; Palmer et al., 2015). This is often motivated in terms of increased sensitivity or efficiency (Srinivasan, Laughlin, and Dubs, 1982; Atick and Redlich, 1990) due to redundancy reduction (Barlow et al., 1961; Barlow, Kaushal, and Mitchison, 1989), i.e. compression. If we consider cortex as a hierarchical latent variable model of sensory inputs, then early sensory areas are implicated in parameterizing the conditional likelihood. The ubiquity of normalization operations in early sensory areas is suggestive of normalization in a flow-based model. That is, early sensory areas may implement the “inference” direction of a flow-based conditional likelihood (Siddharth Agrawal and Dukkipati, 2016; Winkler et al., 2019). This would create a learned, normalized space in which cortex makes predictions. In addition to the sensitivity and efficiency arguments above, this could simplify downstream generative modeling and improve generalization, as demonstrated in Chapter 5. Interestingly, Rao and Ballard, 1999 employ a similar whitening scheme on image inputs, presumably to imitate retina. Normalizing flows offers a generalization of this idea to non-affine parameterizations and multiple stages of normalization. Further, framing early sensory areas in terms of normalizing flows connects these computations conceptually with the rest of the cortical generative model, i.e. truly evaluating the data-level predictions of cortex would require inverting the normalization of early sensory areas. While normalizing flows may not provide a perfect description of early sensory processing (e.g. these operations may not be completely invertible), this framework may help to unify many disparate input normalization circuits. Normalization in Thalamus & Cortex Normalization is also thought to be a key aspect of first-order relays in thalamus, such as the lateral geniculate nucleus (LGN). Dong and Atick, 1995 proposed that inhibition across time in LGN could provide a mechanism for temporally decorrelating the input from retina, with some supporting evidence provided by Dan, Atick, and Reid, 1996. Again, this can 145 retina <latexit sha1_base64="txQdsmSwoi1hWIuSfzTdtrJIzXU=">AAACj3icdVFNSyNBEO2M7qpZdzfRi+Cl2SB4CjO6EE8S9KI3BaNCMoSeTk3S2B9Dd40YhvHXeNX/47+xJ+ZgjBYU/Xj12a+STAqHYfhaC1ZWf/xcW9+o/9r8/edvo7l17UxuOfS4kcbeJsyBFBp6KFDCbWaBqUTCTXJ3WsVv7sE6YfQVTjOIFRtrkQrO0FPDxs4A4QGtKmZvkhYWUGhWlsNGK2yHM6PLIJqDFpnbxbBZGw5GhucKNHLJnOtHYYZxwSwKLqGsD3IHGeN3bAx9DzVT4OJi9oWS7nlmRFNjvWukM/ZjRcGUc1OV+EzFcOI+xyryq1g/x/QoLoTOcgTN3weluaRoaKUHHQkLHOXUA8at8LtSPmGWcfSqLXS6iuKiWq5qszA+UWV9j35kqjUyVA+LeUyOjZ8wUd/Qgi8XZA5yr6oZeQH9SaLPB1gG1wft6LB9cPm/1T2ZH2ed7JJ/ZJ9EpEO65IxckB7h5JE8kWfyEjSDTnAcdN9Tg9q8ZpssWHD+BuvszQE=</latexit> spatiotemporal normalization <latexit sha1_base64="ujduGB53CviLS4rs1Y1/voD4EiU=">AAACl3icdVFNSwMxEE3X7/rV6km8BIvgqeyqoDdFQTwqWBXapWTTaRtMNiGZFcuyV3+NV/0v/huztQdrdSDk8d5MZvImMVI4DMPPSjA3v7C4tLxSXV1b39is1bfunc4shxbXUtvHhDmQIoUWCpTwaCwwlUh4SJ4uS/3hGawTOr3DkYFYsUEq+oIz9FS3RjsIL2hVPr4F5s6UCoIy2jJZFN1aI2yG46CzIJqABpnETbde6XZ6mmcKUuSSOdeOQoNxziwKLqGodjIHhvEnNoC2hylT4OJ8/JWC7numR/va+pMiHbM/K3KmnBupxGcqhkP3WyvJv7R2hv3TOBepyRBS/t2on0mKmpa+0J6wwFGOPGDcCj8r5UNmGUfv3tRLd1Gcl8OVz0y1T1RR3ac/mXIMg+plOo/JgfYdhuofWvDZAuMg867qnjfQryT6vYBZcH/YjI6ah7fHjfOLyXKWyS7ZIwckIifknFyTG9IinLySN/JOPoKd4Cy4Cq6/U4PKpGabTEVw+wXF29DX</latexit> <latexit sha1_base64="OdVnGcyPaciL0UTXCb25xUqGVJA=">AAAClnicdVFNTxsxEHUWaCFACXBB6gGLCIlTtAtI7amKqFB7TCUCSMkq8jqTxMIfK3u2SrraI7+mV/gx/Bu8Sw6EwEiWn96b8YzfJKkUDsPwqRasrK59+ry+Ud/c2v6y09jdu3Ymsxy63EhjbxPmQAoNXRQo4Ta1wFQi4Sa5+1nqN3/BOmH0Fc5SiBUbazESnKGnBo3DPsIUrcqrW2CujVVMin+VXhSDRjNshVXQZRDNQZPMozPYrQ36Q8MzBRq5ZM71ojDFOGcWBZdQ1PuZg5TxOzaGnoeaKXBxXv2koMeeGdKRsf5opBX7uiJnyrmZSnymYjhxb7WSfE/rZTj6HudCpxmC5i+NRpmkaGhpCx0KCxzlzAPGrfCzUj5hlnH05i28dBXFeTlc+cxC+0QV9WP6minHSFFNF/OYHBvfYaI+oAVfLkgdZN5VM/QG+pVEbxewDK5PW9FZ6/TPebN9MV/OOvlKjsgJicg30ia/SYd0CSf35D95II/BQfAjuAx+vaQGtXnNPlmIoPMMsMLQYA==</latexit> cortex <latexit sha1_base64="8BOcPOAGc9bvfisMD6dBL4IsDUc=">AAACj3icdVHLSgMxFE3Hd321uhHcBIvgqsyoUFdSdKM7BatCO5RMeqcNJpMhuSOWYfwat/o//o2Z2oW1euGSw7nPnBulUlj0/c+Kt7C4tLyyulZd39jc2q7Vd+6tzgyHDtdSm8eIWZAigQ4KlPCYGmAqkvAQPV2W8YdnMFbo5A7HKYSKDRMRC87QUf3aXg/hBY3KJ28U51wbh4qiX2v4TX9idB4EU9AgU7vp1yv93kDzTEGCXDJru4GfYpgzg4JLKKq9zELK+BMbQtfBhCmwYT75QkEPHTOgsTbOE6QT9mdFzpS1YxW5TMVwZH/HSvKvWDfD+CzMRZJmCAn/HhRnkqKmpR50IAxwlGMHGDfC7Ur5iBnG0ak20+kuCPNyubLNzPhIFdVD+pMp10hRvczmMTnUbsJI/UMLPl+QWsicqnrgBHQnCX4fYB7cHzeDk+bx7WmjfTE9zirZJwfkiASkRdrkityQDuHklbyRd/Lh1b2Wd+61v1O9yrRml8yYd/0FEUfNEw==</latexit> prediction, error calculation & inference <latexit sha1_base64="vVaZkpB0wXAkjSNsr7BOshRJnYU=">AAAClHicdVFNSwMxEE3X7/pVFbyIECyCBym7KujBgyiCJ1GwKrRLyWanbTDZLMmsWJY9+Wu86q/x35itPVirAyGPN/MykzdRKoVF3/+seFPTM7Nz8wvVxaXlldXa2vq91Znh0ORaavMYMQtSJNBEgRIeUwNMRRIeoqeLMv/wDMYKndzhIIVQsV4iuoIzdFSntt1GeEGj8uEtMHfqWPAyuV8UnVrdb/jDoJMgGIE6GcVNZ63SaceaZwoS5JJZ2wr8FMOcGRRcQlFtZxZSxp9YD1oOJkyBDfPhPwq665iYdrVxJ0E6ZH8qcqasHajIVSqGffs7V5J/5VoZdk/CXCRphpDw70bdTFLUtDSFxsIARzlwgHEj3KyU95lhHJ11Yy/dBWFeDlc+M9Y+UkV1l/5kyjFSVC/jdUz2tOvQV//Qgk8KUguZc1XHzkC3kuD3AibB/UEjOGwc3B7Vz85Hy5knW2SH7JGAHJMzckVuSJNw8kreyDv58Da9U+/Cu/wu9SojzQYZC+/6C+GozyI=</latexit> <latexit sha1_base64="mJJhZ0osDyKosWTLa+p8/zloWpk=">AAACmnicdVHLTsMwEHTDu7wKHOFgUSFxqhJAgiOCC4gLSC0gtVHluJvWwo4je4OoonwAX8MVPoW/wQk9UAp78Wh2dne8G6VSWPT9z5o3N7+wuLS8Ul9dW9/YbGxt31udGQ4drqU2jxGzIEUCHRQo4TE1wFQk4SF6uizzD89grNBJG8cphIoNExELztBR/Uazh/CCRuXVKzAHY7ShnEmeyUpTFE7lt/wq6CwIJqBJJnHb36r1ewPNMwUJcsms7QZ+imHODAouoaj3Mgsp409sCF0HE6bAhnn1m4IeOGZAY+ci1gnSiv1ZkTNl7VhFTqkYjuzvXEn+letmGJ+FuUjSDCHh34PiTFLUtFwNHQgDHOXYAcaNcF4pHzHDOLoFTnVqB2FemivbTI2PVFE/oD+Z0kaK6mVax+RQuwkj9Q8t+GxBaiFzW9UDt0B3kuD3AWbB/VErOG4d3Z00zy8mx1kmu2SfHJKAnJJzckVuSYdw8kreyDv58Pa8C+/au/mWerVJzQ6ZCq/9BTc30do=</latexit> <latexit sha1_base64="jpgwVaaoqN2XA3gRzY6ijcSfzts=">AAACi3icdVHLSgMxFE3HV62v+ti5CRbFVZlRQREXRRFcKlgV2qFk0ts2NJkMyR2xDvMvbvWP/BsztQtr9ULI4dzX4dwokcKi73+WvLn5hcWl8nJlZXVtfaO6ufVgdWo4NLmW2jxFzIIUMTRRoISnxABTkYTHaHhV5B+fwVih43scJRAq1o9FT3CGjupUd9oIL2hUNv4FZu2DPO9Ua37dHwedBcEE1MgkbjubpU67q3mqIEYumbWtwE8wzJhBwSXklXZqIWF8yPrQcjBmCmyYjeXndN8xXdrTxr0Y6Zj92ZExZe1IRa5SMRzY37mC/CvXSrF3FmYiTlKEmH8v6qWSoqaFF7QrDHCUIwcYN8JppXzADOPoHJuadB+EWSGuGDO1PlJ5ZZ/+ZAoZCaqX6Tom+9ptGKh/aMFnGxILqXNVd52B7iTB7wPMgoejenBcP7o7qTUuJ8cpk12yRw5JQE5Jg9yQW9IknLySN/JOPrw179g79y6+S73SpGebTIV3/QU1FMrt</latexit> thalamus (LGN) <latexit sha1_base64="yboAjfk3ClaA8yWc8vSPSAm/nbs=">AAACl3icdVFNSyNBEO2M6+pmv6J7WvbSbBD0EmZ0QW+KwuphWRSMCskQajo1SWP39NBdI4Zhrv4ar/pf/Df2xByM2S0o+vHqs18luZKOwvCpESy9W36/svqh+fHT5y9fW2vrF84UVmBXGGXsVQIOlcywS5IUXuUWQScKL5Prozp+eYPWSZOd0yTHWMMok6kUQJ4atHif8JasLqdvkpY0BgW6cHzzz/HfraoatNphJ5waXwTRDLTZzE4Ha41Bf2hEoTEjocC5XhTmFJdgSQqFVbNfOMxBXMMIex5moNHF5fQrFd/wzJCnxnrPiE/Z1xUlaOcmOvGZGmjs3sZq8l+xXkHpXlzKLC8IM/EyKC0UJ8NrXfhQWhSkJh6AsNLvysUYLAjy6s11Oo/isl6ubjM3PtFVc4O/Zuo1ctK383mgRsZPGOv/0FIsFuQOC6+qGXoB/UmitwdYBBfbnWins332q31wODvOKvvBfrJNFrFddsBO2CnrMsHu2D17YI/B92A/+B2cvKQGjVnNNzZnwdkz9XnPfw==</latexit> <latexit sha1_base64="xgO9wdw3QFH7Y6G0Duse4Qdor5Q=">AAACknicdVHLSgMxFE3Hd33Vx85NsAiuyowKiiu1GxcuFKwK7VAy6Z02mEyG5I5Yhln4NW71c/wbM20X1uqFkMO573OjVAqLvv9V8ebmFxaXlleqq2vrG5u1re0HqzPDocW11OYpYhakSKCFAiU8pQaYiiQ8Rs/N0v/4AsYKndzjMIVQsX4iYsEZOqpb2+sgvKJR+egXmIskBgMJh6Lo1up+wx8ZnQXBBNTJxG67W5Vup6d5piBBLpm17cBPMcyZQcElFNVOZiFl/Jn1oe1gwhTYMB9tUdADx/RorI17CdIR+zMjZ8raoYpcpGI4sL99JfmXr51hfBa6xdIM3WLjRnEmKWpaSkJ7wgBHOXSAcSPcrJQPmGEcnXBTle6DMC+HK8tMtY9UUT2gP5lyjBTV63Qck33tOgzUP7TgswmphcypqntOQHeS4PcBZsHDUSM4bhzdndQvribHWSZ7ZJ8ckoCckgtyTW5Ji3DyRt7JB/n0dr1z79JrjkO9yiRnh0yZd/MNFmjOYA==</latexit> temporal normalization & error calculation <latexit sha1_base64="LDkDIwYFXPUPk9B1QHw2GcGAaK8=">AAACkXicdVFNTxsxEHUW2tKUlgC3crEaReIU7QIS5RaVCxKXVCIfUrKKvM4ksbDXlj2LiFYr9df0Wv4O/wZvuoeEwEgjP7359JvESOEwDJ9rwc7uh4+f9j7Xv+x//XbQODzqO51ZDj2upbbDhDmQIoUeCpQwNBaYSiQMkvvrMj54AOuETu9waSBWbJ6KmeAMPTVpfB8jPKJV+eoVmCMooy2TRTFpNMN2uDK6DaIKNEll3clhbTKeap4pSJFL5twoCg3GObMouISiPs4cGMbv2RxGHqZMgYvz1ScK2vLMlM609Z4iXbHrFTlTzi1V4jMVw4V7HSvJt2KjDGc/41ykJkNI+f9Bs0xS1LRUhE6FBY5y6QHjVvhdKV8wyzh63TY63UVxXi5XttkYn6ii3qLrTLmGQfW4mcfkXPsJC/UOLfh2gXGQeVX11AvoTxK9PsA26J+1o/P22e+LZudXdZw9ckJ+kFMSkUvSITekS3qEkz/kL/lHnoLj4CroBFVuUKtqjsmGBbcvSubOCw==</latexit> <latexit sha1_base64="OdVnGcyPaciL0UTXCb25xUqGVJA=">AAAClnicdVFNTxsxEHUWaCFACXBB6gGLCIlTtAtI7amKqFB7TCUCSMkq8jqTxMIfK3u2SrraI7+mV/gx/Bu8Sw6EwEiWn96b8YzfJKkUDsPwqRasrK59+ry+Ud/c2v6y09jdu3Ymsxy63EhjbxPmQAoNXRQo4Ta1wFQi4Sa5+1nqN3/BOmH0Fc5SiBUbazESnKGnBo3DPsIUrcqrW2CujVVMin+VXhSDRjNshVXQZRDNQZPMozPYrQ36Q8MzBRq5ZM71ojDFOGcWBZdQ1PuZg5TxOzaGnoeaKXBxXv2koMeeGdKRsf5opBX7uiJnyrmZSnymYjhxb7WSfE/rZTj6HudCpxmC5i+NRpmkaGhpCx0KCxzlzAPGrfCzUj5hlnH05i28dBXFeTlc+cxC+0QV9WP6minHSFFNF/OYHBvfYaI+oAVfLkgdZN5VM/QG+pVEbxewDK5PW9FZ6/TPebN9MV/OOvlKjsgJicg30ia/SYd0CSf35D95II/BQfAjuAx+vaQGtXnNPlmIoPMMsMLQYA==</latexit> <latexit sha1_base64="jpgwVaaoqN2XA3gRzY6ijcSfzts=">AAACi3icdVHLSgMxFE3HV62v+ti5CRbFVZlRQREXRRFcKlgV2qFk0ts2NJkMyR2xDvMvbvWP/BsztQtr9ULI4dzX4dwokcKi73+WvLn5hcWl8nJlZXVtfaO6ufVgdWo4NLmW2jxFzIIUMTRRoISnxABTkYTHaHhV5B+fwVih43scJRAq1o9FT3CGjupUd9oIL2hUNv4FZu2DPO9Ua37dHwedBcEE1MgkbjubpU67q3mqIEYumbWtwE8wzJhBwSXklXZqIWF8yPrQcjBmCmyYjeXndN8xXdrTxr0Y6Zj92ZExZe1IRa5SMRzY37mC/CvXSrF3FmYiTlKEmH8v6qWSoqaFF7QrDHCUIwcYN8JppXzADOPoHJuadB+EWSGuGDO1PlJ5ZZ/+ZAoZCaqX6Tom+9ptGKh/aMFnGxILqXNVd52B7iTB7wPMgoejenBcP7o7qTUuJ8cpk12yRw5JQE5Jg9yQW9IknLySN/JOPrw179g79y6+S73SpGebTIV3/QU1FMrt</latexit> <latexit sha1_base64="AsLpb2cB66GdGus9IYFgZ2+JyIU=">AAACjnicdVHLSgMxFE3Hd31VXbhwEyyCqzKjom5E0Y1LhT6EdiiZ9LYNJpMhuSMtw3yNW/0g/8ZM7aK1euGSw7nPnBslUlj0/a+St7S8srq2vlHe3Nre2a3s7TetTg2HBtdSm5eIWZAihgYKlPCSGGAqktCKXh+KeOsNjBU6ruM4gVCxQSz6gjN0VLdy2EEYoVHZ5BWYgTHa5Hm3UvVr/sToIgimoEqm9tTdK3U7Pc1TBTFyyaxtB36CYcYMCi4hL3dSCwnjr2wAbQdjpsCG2eQHOT1xTI/2tXEeI52wsxUZU9aOVeQyFcOh/R0ryL9i7RT712Em4iRFiPnPoH4qKWpayEF7wgBHOXaAcSPcrpQPmWEcnWhznepBmBXLFW3mxkcqL5/QWaZYI0E1ms9jcqDdhKH6hxZ8sSCxkDpVdc8J6E4S/D7AImie1YLz2tnzRfXufnqcdXJEjskpCcgVuSOP5Ik0CCc5eScf5NOreJfejXf7k+qVpjUHZM68x28wWMyz</latexit> <latexit sha1_base64="yVTsLLZdcaXfvUNxxTLh4aMBzW0=">AAAClHicdVFNSwMxEE3X7/pVFbyIECyCp7Krgh48iCJ4EgWrQruUbDptQ5PNksyKZdmTv8ar/hr/jdm1B2t1IOTx3kxm8iZKpLDo+58Vb2Z2bn5hcam6vLK6tl7b2HywOjUcmlxLbZ4iZkGKGJooUMJTYoCpSMJjNLws9MdnMFbo+B5HCYSK9WPRE5yhozq13TbCCxqVlbfAjDPJU1mqed6p1f2GXwadBsEY1Mk4bjsblU67q3mqIEYumbWtwE8wzJhBwSXk1XZqIWF8yPrQcjBmCmyYlf/I6b5jurSnjTsx0pL9WZExZe1IRS5TMRzY31pB/qW1UuydhpmIkxQh5t+NeqmkqGlhCu0KAxzlyAHGjXCzUj5ghnF01k28dB+EWTFc8cxE+0jl1X36kynGSFC9TOYx2deuw0D9Qws+XZBYSJ2ruusMdCsJfi9gGjwcNoKjxuHdcf38YrycRbJD9sgBCcgJOSfX5JY0CSev5I28kw9v2zvzLr2r71SvMq7ZIhPh3XwBSPTPVA==</latexit> 41 Figure 8.8: Computational Schematic of the Visual Pathway. Interpreting the early visual pathway from the perspective of a latent variable model, we can assign computational functions to retina, LGN, and cortex. Retina and LGN are interpreted as implementing normalizing flows, i.e. spatiotemporal predictive coding, reducing spatial and temporal redundancy in the visual input. LGN also serves as the lowest level for hierarchical predictions, which are computed through backward connections in cortex. Using prediction errors throughout the hierarchy, forward cortical connections update latent estimates. be considered as a form of temporal predictive coding (Srinivasan, Laughlin, and Dubs, 1982), removing easily predictable temporal information. Following the interpretation above, normalization in thalamus may provide a second stage of normalizing flow, further removing redundancy. In Chapter 5, we implemented this general technique using autoregressive flows in deep latent variable models. There, we saw that low-level temporal normalization removes static backgrounds, improving modeling and generalization. Further work is needed to assess the functional form (affine/non-affine) and types of dependencies (linear/non-linear) implemented in first-order thalamic relays, though suggestions are given by Dong and Atick, 1995. Normalization, via local lateral inhibition, is also found throughout cortex (King, Zylberberg, and DeWeese, 2013). K. Friston, 2005 suggested that lateral inhibition plays the computational role of inverse covariance (precision) matrices, modeling dependencies between dimensions within the same latent level of the hierarchy. This corresponds to parameterizing approximate posteriors (Rezende and Mohamed, 2015; Kingma, Salimans, et al., 2016) and/or conditional priors (C.-W. Huang et al., 146 2017) with (linear affine) normalizing flows.1 Again, note that normalizing flows offers a more general mathematical framework for describing these normalization computations. Further, predictive coding often assumes that these dependencies are modeled using symmetric lateral weights (K. Friston, 2005). In contrast, normalizing flows also permits non-symmetric schemes, e.g. using spatially autoregressive models (Kingma, Salimans, et al., 2016) or ensembles of such models (Uria, I. Murray, and Larochelle, 2014). In Chapter 5, we also applied normalizing flows across time in hierarchical models to assist in parameterizing dynamics, discussing connections to Friston’s notion of generalized coordinates (K. Friston, 2008a). Thus, multiple forms of spatial and temporal normalization may occur within cortex, allowing cortical columns to add and remove dependencies across space and time. Finally, multiple works within predictive coding have explored the use of prediction precision as a form of attention (Spratling, 2008; Feldman and K. Friston, 2010). Increasing the precision of predictions modulates the gain of prediction errors in driving inference, leading to more precise inferred estimates. This may prove to be a useful technique in machine learning. The overall computational scheme, ignoring spatiotemporal normalization in cortex for simplicity, is shown in Figure 8.8. Motor Dependencies Much like sensory input areas, there is a striking degree of low-level interneuron circuitry in motor output areas. A canonical example is central pattern generator (CPG) circuits, which, through local excitation and inhibition, give rise to coordinated muscle activation (Marder and Bucher, 2001). These circuits provide a basis of “motor primitives,” allowing muscle activations to be carried out in a lower-dimensional manifold rather than the entire combinatorial space. Given the anatomical similarity of these circuits with normalization circuitry in sensory areas, as well as the close relationship between adding (generation) and removing (inference) dependencies in normalizing flows, it is possible that such low-level motor circuits are implementing flow-based distributions. Under this scheme, spinal circuits convert compressed, uncorrelated signals from higher-level motor areas into correlated low-level muscle activation. Similar efficiency arguments from sensory input areas (Atick and Redlich, 1990) are applicable to motor outputs, treating the spinal cord as a communication channel. Further, spatiotemporal dependencies in muscle activation may also improve the expressive capacity and generalization of motor routines. Indeed, multiple works have investigated improving control policies 1 Specifically, Friston’s proposal corresponds to ZCA whitening (K. Friston, 2005), whereas those within machine learning have explored Cholesky whitening (Kingma, Salimans, et al., 2016). 147 normalize <latexit sha1_base64="hGOs4RxcXljv6QdmumAedM2H5c8=">AAACiXicdVHJSgNBEO2Me+IS9eilMQQ8hZkoKJ5ELx4VjArJEHo6laRJL0N3jSQO+RWv+kv+jT0xB2O0oODxanu8SlIpHIbhZylYWV1b39jcKle2d3b3qvsHj85klkOLG2nsc8IcSKGhhQIlPKcWmEokPCWjm6L+9ALWCaMfcJJCrNhAi77gDD3VrR50EMZoVa6NVUyKV5h2q7WwEc6CLoNoDmpkHnfd/VK30zM8U6CRS+ZcOwpTjHNmUXAJ03Inc5AyPmIDaHuomQIX5zPxU1r3TI/2jfWpkc7YnxM5U85NVOI7FcOh+10ryL9q7Qz7F3EudJohaP59qJ9JioYWTtCesMBRTjxg3AqvlfIhs4yj92th00MU54W4Ys3C+URNy3X6kylkpKjGi31MDoy/MFT/0IIvD6QOMu+q6XkD/Uui3w9YBo/NRnTaaN6f1a6u58/ZJEfkmJyQiJyTK3JL7kiLcDImb+SdfASVIAougsvv1qA0nzkkCxHcfAGAjcoy</latexit> un-normalize <latexit sha1_base64="aWZnxFdnt/R05EceVoaYehJUkDk=">AAACjHicdVHJSgNBEO2Me9yi4slLYxC8GGZUUBBBFMSjglEhGUJPp5I06WXorhHjkI/xql/k39gTczBGCwoer7bHqySVwmEYfpaCmdm5+YXFpfLyyuraemVj88GZzHKocyONfUqYAyk01FGghKfUAlOJhMekf1XUH5/BOmH0PQ5SiBXratERnKGnWpXtJsILWpVn+kAbq5gUrzBsVaphLRwFnQbRGFTJOG5bG6VWs214pkAjl8y5RhSmGOfMouAShuVm5iBlvM+60PBQMwUuzkf6h3TPM23aMdanRjpif07kTDk3UInvVAx77netIP+qNTLsnMa50GmGoPn3oU4mKRpamEHbwgJHOfCAcSu8Vsp7zDKO3rKJTfdRnBfiijUT5xM1LO/Rn0whI0X1MtnHZNf4Cz31Dy349EDqIPOumrY30L8k+v2AafBwWIuOaod3x9WLy/FzFskO2SX7JCIn5ILckFtSJ5zk5I28k49gLTgOzoLz79agNJ7ZIhMRXH8BO2LLYA==</latexit> Figure 8.9: Adding & Removing Dependencies with Normalizing Flows. Normalizing flows provides a general mathematical framework for removing (left) or adding (right) probabilistic dependencies. Using lateral interactions, one can move between a normalized (top) or un-normalized (bottom) space. Normalized spaces have benefits for compression, whereas un-normalized spaces are more expressive. Neural systems may employ these transforms for sensory and motor processing. with normalizing flows across motor dimensions (Haarnoja et al., 2018; Tang and Shipra Agrawal, 2018; Ward, Smofsky, and Bose, 2019), and in Chapter 7, we investigated incorporating flow-based motor dependencies across time. Consistent with this setup, it has been empirically observed in neural systems that higher-level motor areas output transient update signals, which are converted to lower-level motor trajectories (Shalit et al., 2012). While these findings are encouraging, more apparent benefits are likely to arise in high-dimensional action spaces, where the curse of dimensionality (Chen and Gopinath, 2001) makes it essential to model dependencies for effective exploration. For instance, current MuJoCo environments from OpenAI gym (Todorov, Erez, and Tassa, 2012; Brockman et al., 2016) contain <20 action dimensions, whereas it has been estimated that there are roughly 800 independent dimensions to human motor control (Powers, 1973). Operating and learning efficiently in such large action spaces may require modeling spatiotemporal dependencies through some form of hierarchical or flow-based decomposition. 148 8.5 Discussion In this chapter, we connected the ideas presented in this thesis back to their origins in predictive coding. The two core techniques developed in this thesis, iterative amortized inference and sequential autoregressive flows, map onto aspects of hierarchical and spatiotemporal predictive coding, respectively. By connecting these techniques back to predictive coding, we arrived at a variety of possible implied correspondences between machine learning and neuroscience. In particular, • we identified the dendrites of pyramidal neurons as functionally analogous to (nonlinear) deep networks, and • we identified lateral inhibition as implementing normalizing flows. Placing pyramidal neuron dendrites in correspondence with deep networks departs from the traditional one-to-one analogy of biological and artificial neurons, raising a host of questions regarding dendritic computation and learning via backpropagation. Likewise, normalizing flows offers a more general framework for considering the normalization computations carried out by lateral inhibitory interactions found within multiple brain regions. We are hopeful that connecting these areas will provide new insights for both machine learning and neuroscience. For practical reasons, we primarily focused on perception in discussing predictive coding and the ideas in this thesis. This is a result of the fact that predictive coding was initially developed and studied in the context of generative models of sensory inputs (Srinivasan, Laughlin, and Dubs, 1982; Rao and Ballard, 1999; K. Friston, 2005). However, if the cortical microcircuit implements a general-purpose modeling and inference algorithm, we should expect similar computations to be applicable to motor and prefrontal cortices. Over the past decade, Friston and colleagues have developed a range of exciting ideas, interpreting motor control as a process of proprioceptive prediction (Adams, Shipp, and K. J. Friston, 2013) and prefrontal cortex as performing hierarchical goal inference (Pezzulo, Rigoli, and K. J. Friston, 2018). Such ideas can be seen as modern extensions of early ideas in cybernetics (Wiener, 1948; MacKay, 1956; Powers, 1973), using motor control to correct for discrepancies (errors) between desired and actual outcomes. Indeed, as shown in Part III, the same perceptual modeling and inference techniques can be applied to control. Nevertheless, as we discuss in the final chapter, further work is needed within machine learning to unify perception and control under a single formulation. 149 References Adams, Rick A, Stewart Shipp, and Karl J Friston (2013). “Predictions not commands: active inference in the motor system”. In: Brain Structure and Function 218.3, pp. 611–643. Agrawal, Siddharth and Ambedkar Dukkipati (2016). “Deep Variational Inference Without Pixel-Wise Reconstruction”. In: arXiv preprint arXiv:1611.05209. Alink, Arjen et al. (2010). “Stimulus predictability reduces responses in primary visual cortex”. In: Journal of Neuroscience 30.8, pp. 2960–2966. Atal, B and M Schroeder (1979). “Predictive coding of speech signals and subjective error criteria”. In: IEEE Transactions on Acoustics, Speech, and Signal Processing 27.3, pp. 247–254. Atick, Joseph J and A Norman Redlich (1990). “Towards a theory of early visual processing”. In: Neural computation 2.3, pp. 308–320. – (1992). “What does the retina know about natural scenes?” In: Neural computation 4.2, pp. 196–210. Baccus, Stephen A et al. (2008). “A retinal circuit that computes object motion”. In: Journal of Neuroscience 28.27, pp. 6807–6817. Barlow, Horace B et al. (1961). “Possible principles underlying the transformation of sensory messages”. In: Sensory communication 1, pp. 217–234. Barlow, Horace B, Tej P Kaushal, and Graeme J Mitchison (1989). “Finding minimum entropy codes”. In: Neural Computation 1.3, pp. 412–423. Bastos, Andre Moraes, W Martin Usrey, et al. (2012). “Canonical microcircuits for predictive coding”. In: Neuron 76.4, pp. 695–711. Bastos, Andre Moraes, Julien Vezoli, et al. (2015). “Visual areas exert feedforward and feedback influences through distinct frequency channels”. In: Neuron 85.2, pp. 390–401. Bekkers, John M (2011). “Pyramidal neurons”. In: Current Biology 21.24, R975. Bell, Curtis C (2001). “Memory-based expectations in electrosensory systems”. In: Current opinion in neurobiology 11.4, pp. 481–487. Bengio, Yoshua (2014). “How auto-encoders could provide credit assignment in deep networks via target propagation”. In: arXiv preprint arXiv:1407.7906. Bi, Guo-qiang and Mu-ming Poo (1998). “Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type”. In: Journal of neuroscience 18.24, pp. 10464–10472. Brockman, Greg et al. (2016). “Openai gym”. In: arXiv preprint arXiv:1606.01540. Broeke, Gerben van den (2016). “What auto-encoders could learn from brains”. MA thesis. Aalto University. 150 Brunner, János and János Szabadics (2016). “Analogue modulation of back-propagating action potentials enables dendritic hybrid signalling”. In: Nature communications 7, p. 13033. Carandini, Matteo and David J Heeger (2012). “Normalization as a canonical neural computation”. In: Nature Reviews Neuroscience 13.1, pp. 51–62. Chen, Scott Saobing and Ramesh A Gopinath (2001). “Gaussianization”. In: Advances in neural information processing systems, pp. 423–429. Clark, Andy (2013). “Whatever next? Predictive brains, situated agents, and the future of cognitive science”. In: Behavioral and Brain Sciences 36.3, pp. 181–204. Covic, Elise N and S Murray Sherman (2011). “Synaptic properties of connections between the primary and secondary auditory cortices in mice”. In: Cerebral Cortex 21.11, pp. 2425–2441. Crick, Francis (1989). “The recent excitement about neural networks”. In: Nature 337.6203, pp. 129–132. Dan, Yang, Joseph J Atick, and R Clay Reid (1996). “Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory”. In: Journal of Neuroscience 16.10, pp. 3351–3362. Dasgupta, Ishita et al. (2018). “Remembrance of inferences past: Amortization in human hypothesis generation”. In: Cognition 178, pp. 67–81. Dayan, Peter et al. (1995). “The helmholtz machine”. In: Neural computation 7.5, pp. 889–904. De Pasquale, Roberto and S Murray Sherman (2011). “Synaptic properties of corticocortical connections between the primary and secondary visual cortical areas in the mouse”. In: Journal of Neuroscience 31.46, pp. 16494–16506. Dinh, Laurent, Jascha Sohl-Dickstein, and Samy Bengio (2017). “Density estimation using real nvp”. In: International Conference on Learning Representations. Dong, Dawei W and Joseph J Atick (1995). “Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus”. In: Network: Computation in Neural Systems 6.2, pp. 159–178. Douglas, Rodney J, Kevan AC Martin, and David Whitteridge (1989). “A canonical microcircuit for neocortex”. In: Neural computation 1.4, pp. 480–488. Durkan, Conor et al. (2019). “Neural Spline Flows”. In: arXiv preprint arXiv:1906.04032. Egner, Tobias, Jim M Monti, and Christopher Summerfield (2010). “Expectation and surprise determine neural population responses in the ventral visual stream”. In: Journal of Neuroscience 30.49, pp. 16601–16608. Eliades, Steven J and Xiaoqin Wang (2008). “Neural substrates of vocalization feedback monitoring in primate auditory cortex”. In: Nature 453.7198, p. 1102. 151 Feldman, Harriet and Karl Friston (2010). “Attention, uncertainty, and free-energy”. In: Frontiers in human neuroscience 4. Friedman, Jerome H (1987). “Exploratory projection pursuit”. In: Journal of the American statistical association 82.397, pp. 249–266. Friston, Karl (2005). “A theory of cortical responses”. In: Philosophical Transactions of the Royal Society of London B: Biological Sciences 360.1456, pp. 815–836. – (2008a). “Hierarchical models in the brain”. In: PLoS computational biology 4.11, e1000211. – (2008b). “Variational filtering”. In: NeuroImage 41.3, pp. 747–766. – (2018). “Does predictive coding have a future?” In: Nature neuroscience 21.8, p. 1019. Friston, Karl et al. (2007). “Variational free energy and the Laplace approximation”. In: Neuroimage 34.1, pp. 220–234. Fujioka, Takako et al. (2009). “Beta and gamma rhythms in human auditory cortex during musical beat processing”. In: Annals of the New York Academy of Sciences 1169.1, pp. 89–92. Gershman, Samuel J (2019). “What does the free energy principle tell us about the brain?” In: arXiv preprint arXiv:1901.07945. Gidon, Albert et al. (2020). “Dendritic action potentials and computation in human layer 2/3 cortical neurons”. In: Science. Gilbert, Charles D and Mariano Sigman (2007). “Brain states: top-down influences in sensory processing”. In: Neuron 54.5, pp. 677–696. Girard, Pascal and Jean Bullier (1989). “Visual activity in area V2 during reversible inactivation of area 17 in the macaque monkey”. In: Journal of neurophysiology 62.6, pp. 1287–1302. Girard, Pascal, PA Salin, and Jean Bullier (1991). “Visual activity in areas V3a and V3 during reversible inactivation of area V1 in the macaque monkey”. In: Journal of Neurophysiology 66.5, pp. 1493–1503. Graham, Daniel J, Damon M Chandler, and David J Field (2006). “Can the theory of “whitening” explain the center-surround properties of retinal ganglion cell receptive fields?” In: Vision research 46.18, pp. 2901–2913. Grossberg, Stephen (1987). “Competitive learning: From interactive activation to adaptive resonance”. In: Cognitive science 11.1, pp. 23–63. Guergiuev, Jordan, Timothy P Lillicrap, and Blake A Richards (2016). “Biologically feasible deep learning with segregated dendrites”. In: arXiv preprint arXiv:1610.00161. Haarnoja, Tuomas et al. (2018). “Latent Space Policies for Hierarchical Reinforcement Learning”. In: International Conference on Machine Learning, pp. 1846–1855. 152 Harrison, CW (1952). “Experiments with linear prediction in television”. In: Bell System Technical Journal 31.4, pp. 764–783. Hawkins, Jeff and Sandra Blakeslee (2004). On intelligence: How a new understanding of the brain will lead to the creation of truly intelligent machines. Macmillan. Hebb, Donald O (1949). “The organization of behavior; a neuropsycholocigal theory.” In: A Wiley Book in Clinical Psychology., pp. 62–78. Hinton, Geoffrey E. (2007). “How to do backpropagation in a brain”. In: NeurIPS Deep Learning Workshop. Hosoya, Toshihiko, Stephen A Baccus, and Markus Meister (2005). “Dynamic predictive coding by the retina”. In: Nature 436.7047, p. 71. Huang, Chin-Wei et al. (2017). “Learnable explicit density for continuous latent space and variational inference”. In: arXiv preprint arXiv:1710.02248. Huang, Yanping and Rajesh PN Rao (2011). “Predictive coding”. In: Wiley Interdisciplinary Reviews: Cognitive Science 2.5, pp. 580–593. Isaacson, Jeffry S and Massimo Scanziani (2011). “How inhibition shapes cortical activity”. In: Neuron 72.2, pp. 231–243. Ito, Masao (1998). “Cerebellar learning in the vestibulo–ocular reflex”. In: Trends in cognitive sciences 2.9, pp. 313–321. Jehee, Janneke FM and Dana H Ballard (2009). “Predictive feedback can account for biphasic responses in the lateral geniculate nucleus”. In: PLoS Comput Biol 5.5, e1000373. Johenning, Friedrich W et al. (2015). “Ryanodine receptor activation induces longterm plasticity of spine calcium dynamics”. In: PLoS biology 13.6, e1002181. Kalman, Rudolph Emil (1960). “A new approach to linear filtering and prediction problems”. In: Journal of Basic Engineering 82.1, pp. 35–45. Keller, Georg B, Tobias Bonhoeffer, and Mark Hübener (2012). “Sensorimotor mismatch signals in primary visual cortex of the behaving mouse”. In: Neuron 74.5, pp. 809–815. Keller, Georg B and Thomas D Mrsic-Flogel (2018). “Predictive processing: a canonical cortical computation”. In: Neuron 100.2, pp. 424–435. Kennedy, Ann et al. (2014). “A temporal basis for predicting the sensory consequences of motor commands in an electric fish”. In: Nature neuroscience 17.3, pp. 416–422. Kessy, Agnan, Alex Lewin, and Korbinian Strimmer (2018). “Optimal whitening and decorrelation”. In: The American Statistician 72.4, pp. 309–314. King, Paul D, Joel Zylberberg, and Michael R DeWeese (2013). “Inhibitory interneurons decorrelate excitatory cells to drive sparse code formation in a spiking model of V1”. In: Journal of Neuroscience. 153 Kingma, Durk P, Tim Salimans, et al. (2016). “Improved variational inference with inverse autoregressive flow”. In: Advances in neural information processing systems, pp. 4743–4751. Kingma, Durk P and Max Welling (2014). “Stochastic gradient VB and the variational auto-encoder”. In: Proceedings of the International Conference on Learning Representations. Körding, Konrad P and Peter König (2001). “Supervised and unsupervised learning with two sites of synaptic integration”. In: Journal of computational neuroscience 11.3, pp. 207–215. Kumar, Manoj et al. (2020). “VideoFlow: A Flow-Based Generative Model for Video”. In: International Conference on Learning Representations. Lee, Dong-Hyun et al. (2015). “Difference target propagation”. In: Joint european conference on machine learning and knowledge discovery in databases. Springer, pp. 498–515. Lillicrap, Timothy P, Daniel Cownden, et al. (2016). “Random synaptic feedback weights support error backpropagation for deep learning”. In: Nature communications 7, p. 13276. Lillicrap, Timothy P, Adam Santoro, et al. (2020). “Backpropagation and the brain”. In: Nature Reviews Neuroscience, pp. 1–12. London, Michael and Michael Häusser (2005). “Dendritic computation”. In: Annu. Rev. Neurosci. 28, pp. 503–532. Lotter, William, Gabriel Kreiman, and David Cox (2018). “A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception”. In: arXiv preprint arXiv:1805.10734. MacKay, D M (1956). “The epistemological problem for automata”. In: Automata studies, pp. 235–252. Marder, Eve and Dirk Bucher (2001). “Central pattern generators and the control of rhythmic movements”. In: Current biology. Markram, Henry et al. (1997). “Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs”. In: Science 275.5297, pp. 213–215. McCulloch, Warren S and Walter Pitts (1943). “A logical calculus of the ideas immanent in nervous activity”. In: The bulletin of mathematical biophysics 5.4, pp. 115–133. Mel, Bartlett W (1992). “The clusteron: toward a simple abstraction for a complex neuron”. In: Advances in neural information processing systems. Meyer, Hanno S et al. (2011). “Inhibitory interneurons in a cortical column form hot zones of inhibition in layers 2 and 5A”. In: Proceedings of the National Academy of Sciences 108.40, pp. 16807–16812. 154 Meyer, Travis and Carl R Olson (2011). “Statistical learning of visual transitions in monkey inferotemporal cortex”. In: Proceedings of the National Academy of Sciences 108.48, pp. 19401–19406. Minsky, Marvin and Seymour Papert (1969). Perceptrons. Mountcastle, VB, AL Berman, and PW Davies (1955). “Topographic organization and modality representation in first somatic area of cat’s cerebral cortex by method of single unit analysis”. In: Am J Physiol 183.464, p. 10. Mumford, David (1992). “On the computational architecture of the neocortex”. In: Biological cybernetics 66.3, pp. 241–251. Murray, Scott O et al. (2002). “Shape perception reduces activity in human primary visual cortex”. In: Proceedings of the National Academy of Sciences 99.23, pp. 15164–15169. Oliver, BM (1952). “Efficient coding”. In: The Bell System Technical Journal 31.4, pp. 724–750. Ölveczky, Bence P, Stephen A Baccus, and Markus Meister (2003). “Segregation of object and background motion in the retina”. In: Nature 423.6938, pp. 401–408. Palmer, Stephanie E et al. (2015). “Predictive information in a sensory population”. In: Proceedings of the National Academy of Sciences 112.22, pp. 6908–6913. Parras, Gloria G et al. (2017). “Neurons along the auditory pathway exhibit a hierarchical organization of prediction error”. In: Nature communications 8.1, pp. 1–17. Pezzulo, Giovanni, Francesco Rigoli, and Karl J Friston (2018). “Hierarchical active inference: A theory of motivated control”. In: Trends in cognitive sciences 22.4, pp. 294–306. Pitkow, Xaq and Markus Meister (2012). “Decorrelation and efficient coding by retinal ganglion cells”. In: Nature neuroscience 15.4, p. 628. Poirazi, Panayiota, Terrence Brannon, and Bartlett W Mel (2003). “Pyramidal neuron as two-layer neural network”. In: Neuron 37.6, pp. 989–999. Polsky, Alon, Bartlett W Mel, and Jackie Schiller (2004). “Computational subunits in thin dendrites of pyramidal cells”. In: Nature neuroscience 7.6, p. 621. Pourahmadi, Mohsen (2011). “Covariance estimation: The GLM and regularization perspectives”. In: Statistical Science, pp. 369–387. Powers, William T (1973). Behavior: The control of perception. Aldine Chicago. Rao, Rajesh PN (1998). “Correlates of attention in a model of dynamic visual recognition”. In: Advances in neural information processing systems, pp. 80–86. Rao, Rajesh PN and Dana H Ballard (1999). “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” In: Nature neuroscience 2.1. 155 Rao, Rajesh PN and Terrence J Sejnowski (2002). “Predictive Coding, Cortical Feedback, and Spike-Timing Dependent Plasticity”. In: Probabilistic models of the brain, p. 297. Rezende, Danilo Jimenez and Shakir Mohamed (2015). “Variational Inference with Normalizing Flows”. In: International Conference on Machine Learning, pp. 1530–1538. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra (2014). “Stochastic backpropagation and approximate inference in deep generative models”. In: Proceedings of the International Conference on Machine Learning, pp. 1278– 1286. Richards, Blake A (Sept. 6, 2019). personal communication. Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). “Learning representations by back-propagating errors.” In: Nature. Schiess, Mathieu, Robert Urbanczik, and Walter Senn (2016). “Somato-dendritic synaptic plasticity and error-backpropagation in active dendrites”. In: PLoS computational biology 12.2, e1004638. Shalit, Uri et al. (2012). “Descending systems translate transient cortical commands into a sustained muscle activation signal”. In: Cerebral cortex 22.8, pp. 1904–1914. Shannon, Claude E (1948). “A mathematical theory of communication”. In: The Bell system technical journal 27.3, pp. 379–423. Sharma, Jitendra, Alessandra Angelucci, and Mriganka Sur (2000). “Induction of visual orientation modules in auditory cortex”. In: Nature 404.6780, p. 841. Sherman, S Murray and RW Guillery (2002). “The role of the thalamus in the flow of information to the cortex”. In: Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 357.1428, pp. 1695–1708. Spratling, Michael W (2008). “Reconciling predictive coding and biased competition models of cortical function”. In: Frontiers in computational neuroscience 2, p. 4. Srinivasan, Mandyam Veerambudi, Simon Laughlin, and Andreas Dubs (1982). “Predictive coding: a fresh view of inhibition in the retina”. In: Proceedings of the Royal Society of London. Series B. Biological Sciences 216.1205, pp. 427–459. Stork, David G (1989). “Is backpropagation biologically plausible”. In: International Joint Conference on Neural Networks. Vol. 2. IEEE Washington, DC, pp. 241–246. Stuart, Greg J and Bert Sakmann (1994). “Active propagation of somatic action potentials into neocortical pyramidal cell dendrites”. In: Nature 367.6458, p. 69. Stuart, Greg et al. (1997). “Action potential initiation and backpropagation in neurons of the mammalian CNS”. In: Trends in neurosciences 20.3, pp. 125–131. Summerfield, Christopher et al. (2006). “Predictive codes for forthcoming perception in the frontal cortex”. In: Science 314.5803, pp. 1311–1314. 156 Tang, Yunhao and Shipra Agrawal (2018). “Boosting Trust Region Policy Optimization by Normalizing Flows Policy”. In: arXiv preprint arXiv:1809.10326. Todorov, Emanuel, Tom Erez, and Yuval Tassa (2012). “Mujoco: A physics engine for model-based control”. In: Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, pp. 5026–5033. Uria, Benigno, Iain Murray, and Hugo Larochelle (2014). “A deep and tractable density estimator”. In: International Conference on Machine Learning, pp. 467– 475. Von Helmholtz, Hermann (1867). Handbuch der physiologischen Optik. Vol. 9. Voss. Wacongne, Catherine et al. (2011). “Evidence for a hierarchy of predictions and prediction errors in human cortex”. In: Proceedings of the National Academy of Sciences 108.51, pp. 20754–20759. Walsh, Kevin S et al. (2020). “Evaluating the neurophysiological evidence for predictive processing as a model of perception”. In: Annals of the New York Academy of Sciences 1464.1, p. 242. Ward, Patrick Nadeem, Ariella Smofsky, and Avishek Joey Bose (2019). “Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies”. In: ICML Workshop on Invertible Neural Nets and Normalizing Flows. Werbos, Paul (1974). “Beyond regression:" new tools for prediction and analysis in the behavioral sciences”. In: Ph. D. dissertation, Harvard University. Whittington, James CR and Rafal Bogacz (2017). “An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity”. In: Neural computation. Wiegand, Thomas et al. (2003). “Overview of the H. 264/AVC video coding standard”. In: IEEE Transactions on circuits and systems for video technology 13.7, pp. 560– 576. Wiener, Norbert (1948). Cybernetics or Control and Communication in the Animal and the Machine. MIT press. Williams, Stephen R and Greg J Stuart (2000). “Backpropagation of physiological spike trains in neocortical pyramidal neurons: implications for temporal coding in dendrites”. In: Journal of Neuroscience. Winkler, Christina et al. (2019). “Learning Likelihoods with Conditional Normalizing Flows”. In: arXiv preprint arXiv:1912.00042. Xie, Xiaohui and H Sebastian Seung (2003). “Equivalence of backpropagation and contrastive Hebbian learning in a layered network”. In: Neural computation 15.2, pp. 441–454. 157 Yildirim, Ilker et al. (2015). “Efficient and robust analysis-by-synthesis in vision: A computational framework, behavioral tests, and modeling neuronal representations”. In: Thirty-Seventh Annual Conference of the Cognitive Science Society. Vol. 4. Zador, Anthony M, Brenda J Claiborne, and Thomas H Brown (1992). “Nonlinear pattern separation in single hippocampal neurons with active dendritic membrane”. In: Advances in neural information processing systems, pp. 51–58. Zmarz, Pawel and Georg B Keller (2016). “Mismatch receptive fields in mouse visual cortex”. In: Neuron 92.4, pp. 766–772. 158 Chapter 9 CONCLUSION Marino, Joseph and Yisong Yue (2019). “An Inference Perspective on Model-Based Reinforcement Learning”. In: ICML Workshop on Generative Modeling and Model-Based Reasoning for Robotics and AI. url: https://drive.google. com/open?id=1uc1FdjHVkkEAaU6jz7aVrP3khGyqkdKi. This thesis has presented two core techniques for probabilistic modeling and inference, iterative amortization and sequential autoregressive flows, applied to both perception (Part II) and control (Part III). These techniques are modern incarnations of classical feedback and feedforward ideas popularized and developed under cybernetics (Wiener, 1948; Ashby, 1956). Importantly, these techniques are both learned, providing an additional layer of optimization, i.e., negative feedback, for added efficiency and efficacy. Thus, these ideas are both generally applicable to a range of perception and control settings, as well as powerful, enabling performance improvements. Figure 9.1 provides an overview of the contributions. In this chapter, we conclude this thesis by examining the general themes developed in the preceding chapters and broadening the discussion toward a synthesis of these ideas and the path ahead. 9.1 Iterative Estimation The technical contributions of this thesis demonstrate the benefits of iterative estimation, applied to both probabilistic modeling of dynamics (sequential autoregressive flows; Chapters 5 & 7) and variational inference (iterative amortization; Chapters 3, 4, & 6). We have already seen how iterative amortization provides updates to inferred estimates of distribution parameters, λ: λ ← 𝑓𝜙 (λ, ∇λ L). In each case, we used a gated transform to parameterize this update, which is a specialized version of a residual update: λ ← λ + Δλ. (9.1) This had numerous benefits, including the ability to improve performance by simply increasing the number of iterations, i.e., amount of computation, and the 159 (a) Ch. 3 (b) Ch. 4 (c) Ch. 5 13 10 9 (d) Ch. 6 (e) Ch. 7 31 Figure 9.1: Thesis Summary. Graphical representations of perception (a–c) and control (d–e) using feedback (a, b, d) and feedforward (c, e) processes. 14 ability to estimate multiple solutions (Greff et al., 2019). Iterative estimation also allowed us to automatically adapt to new priors, as seen in sequential latent variable models (Chapter 4), and new conditional likelihoods, as seen in policy optimization (Chapter 6). Thus, this procedure is both more powerful and more general. Switching settings, with sequential autoregressive flows on observations, x1:𝑇 , we can consider the special case of parameterizing the shift at time 𝑡 as x𝑡−1 and the scale as the vector 1. The normalized base distribution variable then becomes x𝑡 − x𝑡−1 , 1 ≡ Δx𝑡−1 . y𝑡 = Equivalently, if we are modeling y𝑡 = Δx𝑡−1 using a higher-level model for the base distribution, the autoregressive generative model can be expressed as x𝑡 ← x𝑡−1 + Δx𝑡−1 . (9.2) In Chapters 5 & 7, we saw that this scheme simplified estimation, improving performance in both cases. With conditional likelihood estimation (Chapter 5), we also saw accompanying generalization improvements, resulting from the learned normalization of inputs. We can interpret x𝑡−1 in Eq. 9.2 as a type of frame of reference, providing an initial estimate of x𝑡 , which is then refined through Δx𝑡−1 . 160 While formulated from the differing perspectives of feedback and feedforward processing, we see in Eqs. 9.1 & 9.2 that iterative amortization and sequential autoregressive flows ultimately involve similar iterative estimation schemes, estimating changes from previous estimates. Iterative estimation is arguably the most generic scheme for both optimization and prediction, as it simply asks, “should this variable be larger or smaller?” In cases where variables change relatively smoothly, due to the continuity of time or functional constraints, this is a useful inductive bias for performing estimation. As we have shown in this thesis, the same mathematical principles can be applied to both perception (Part II) and control (Part III), highlighting the generality of these iterative estimation schemes. This suggests a unified set of relatively simple computational principles for performing a host of processes, an alluring possibility given the seeming standardization of biological neural “hardware.” Thus, considering the biological connections discussed in Chapter 8, top-down and bottom-up cortical projections (mediated by pyramidal neurons) may use similar computational principles, respectively computing changes in predictions and latent estimates. 9.2 Combining Generative Perception & Control As we have laid out each of these ideas separately, there is still work to be done in combining these techniques to form a unified agent, combining both perception and control. Importantly, this is not as simple as merely stitching together the computation graphs for perception and control, e.g., Figures 9.1b & 9.1d. We can see this by starting from the control-as-inference perspective (Levine, 2018), presented in Chapter 6, and considering a hierarchical policy, with an internal latent variable, z. Then, we can express the agent-environment interaction1 as 𝑝 𝜃 (s1:𝑇 , z1:𝑇 , a1:𝑇 ) = 𝜌(s1 ) 𝑇 Ö 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 ) 𝑝 𝜃 (z𝑡 |s≤𝑡 , z<𝑡 , a<𝑡 ) 𝑝 𝜃 (a𝑡 |s≤𝑡 , z≤𝑡 , a<𝑡 ). 𝑡=1 Note that the distribution over z, i.e., 𝑝 𝜃 (z𝑡 |s≤𝑡 , z<𝑡 , a<𝑡 ), is expressed as a direct conditional mapping, rather than as an inverse generative mapping (as in Part II). This is a result of the mathematical formulation of policy distributions and not a design choice. However, we can use Bayes’ rule to re-express the posterior in terms of a generative mapping 𝑝 𝜃 (z𝑡 |s≤𝑡 , z<𝑡 , a<𝑡 ) = 𝑝 𝜃 (s𝑡 |s<𝑡 , z≤𝑡 , a<𝑡 ) 𝑝 𝜃 (z𝑡 |s<𝑡 , z<𝑡 , a<𝑡 ) . 𝑝 𝜃 (s𝑡 |s<𝑡 , z<𝑡 , a<𝑡 ) (9.3) 1 To simplify notation, we express the environment in terms of states, i.e., an MDP. However, an additional mapping from states to observations could be used, yielding a POMDP, as in Marino and Yue, 2019. 161 Much like the perception generative models from Part II, the numerator in Eq. 9.3 is the agent’s action-conditioned generative model, a joint distribution over internal latent variables and state observations. However, we see that the denominator is the marginal likelihood of the current observation, conditioned only on the past variables. To see the implications of this term, we can continue with the controlas-inference approach, introducing an approximate posterior, 𝜋(𝜏|O), now over 𝜏 ≡ [s1:𝑇 , z1:𝑇 , a1:𝑇 ]. At time step 𝑡, we have 𝑝 env (s𝑡+1 |s𝑡 , a𝑡 )𝜋(z𝑡 |s≤𝑡 , z<𝑡 , a<𝑡 , O)𝜋(a𝑡 |s≤𝑡 , z≤𝑡 , a<𝑡 , O), where, as before, O denotes optimality. Plugging this into the variational objective (and ignoring Lagrange factors for simplicity), we then have Task Information Gain Internal KL }| { z }| { " 𝑇 reward z Õ z}|{ 𝑝 𝜃 (s𝑡 |s<𝑡 , z≤𝑡 , a<𝑡 ) 𝜋(z𝑡 |s≤𝑡 , z<𝑡 , a<𝑡 , O) J = E 𝑝env ,𝜋 𝑟 𝑡 + log − log (9.4) 𝑝 𝜃 (s𝑡 |s<𝑡 , z<𝑡 , a<𝑡 ) 𝑝 𝜃 (z𝑡 |s<𝑡 , z<𝑡 , a<𝑡 ) 𝑡=1 # 𝜋(a𝑡 |s≤𝑡 , z≤𝑡 , a<𝑡 , O) − log . 𝑝 𝜃 (a𝑡 |s≤𝑡 , z≤𝑡 , a<𝑡 ) | {z } Action KL The first and last term are the typical reward and action KL from the control-asinference formulation, modified for the hierarchical setting. The “Internal KL” term resembles the latent dynamics found within the generative models from Part II. However, instead of just the conditional log-likelihood of the current state observation, s𝑡 , we have a “Task Information Gain” term. This is the crucial difference from simply combining the perception and control techniques from this thesis, which would only yield the numerator in this term. Briefly, because z is sampled from 𝜋, which is optimized for the task (i.e., conditioned on O), the information gain can be thought of as the amount of task-relevant information extracted from the current state observation. In other words, the agent is optimized to learn an internal model that maximally extracts task-relevant information from the environment. This contrasts with current approaches to model-based reinforcement learning with latent variable models (Ha and Schmidhuber, 2018; Hafner et al., 2019), which learn to model observations as well as possible, i.e., the simple combination of perception and control techniques. Intuitively, by modeling everything, the agent’s model may neglect less salient task-relevant information, resulting in poor performance. Similarly, from an evolutionary perspective, organisms obtain no utility in modeling their environment unless it can extract information relevant for survival and reproduction. 162 Figure 9.2: Combining Generative Perception & Control. Graphical representation combining generative perception17(Part II) and control (Part III). Internal perceptual latent variables are a hierarchical variable in the control policy. Importantly, the agent maximizes the task-relevant information gain from the environment’s state observation, rather than the conditional log-likelihood. Stepping back, this exercise also demonstrates that internal perceptual variables may be reasonably considered as hierarchical variables in the control policy. This echoes the “motor chauvinist” view from neuroscience (Wolpert, Ghahramani, and Flanagan, 2001): the brain is ultimately concerned with control, with perception simply a means of improving control. Interestingly, if we consider deeper hierarchies of variables, there may not be a clear line between perception and control, supporting the application of similar computational principles put forth in this thesis. For a more in-depth derivation of this formulation, we refer to Marino and Yue, 2019. In this section, we have attempted to combine perception and control by treating perception as an initial stage of processing, providing an input to the control policy. While we considered systems with only a single internal latent variable for perception, one could, in theory, construct policies with many hierarchical levels. However, even with many variables, this scheme is ultimately just a method for parameterizing more complex policy distributions, interpreting intermediate variables as “perceptual variables.” In this sense, we have a hierarchy of actions. In the next section, we discuss an alternative approach toward combining perception and control, instead using a hierarchy of tasks. 163 9.3 Controlling Perception In a task hierarchy, each level receives a task (e.g., a goal) from the level above, defined in the space of the perceptual representation at the current level. The level’s “policy” attempts to satisfy this task by assigning tasks to the level below. In this way, each level attempts to control its perception (Powers, 1973), bringing the perceptual state estimate toward the goal. The top level assigns sub-tasks to solve the overall task, e.g., survival, and the lowest level outputs primitive actions to the environment. By estimating and controlling perceptions at each level, the agent can control abstract aspects of the environment, simplifying the process of solving the overall task over larger spatiotemporal scales. To a greater degree than in an action hierarchy, perception and control are intimately linked in this formulation. That is, perception does not merely precede control. Rather, the two processes are concurrent, with perception providing the basis for estimating and evaluating control. In the RL literature, this setup is sometimes referred to as feudal RL (Dayan and Hinton, 1993), with several recent incarnations (Vezhnevets et al., 2017; Nachum et al., 2018a). In the control theory literature, it is sometimes referred to as hierarchical or cascaded control (Albus, Barbera, and Nagel, 1980). And in the cognitive science and neuroscience literature, this relates to ideas of hierarchical cognitive control (Badre, 2008; Yin, 2016). We now discuss several aspects of this approach, as well as connections to concepts across various fields. Task Specification with State Distributions We require some method for defining the tasks throughout the hierarchy. To define these tasks, we first step back and reconsider the general framing of tasks. In Part III, we utilized the control-as-inference formulation of reinforcement learning, largely following the notation of Levine, 2018. This formulation, born out of an effort to frame reinforcement learning (Sutton and Barto, 2018) in terms of probabilistic inference, makes the somewhat unnatural move of introducing auxiliary “optimality” variables (Cooper, 1988), O, which are set to 1 by construction. While this mathematical trick provides a convenient framing of the policy optimization problem in RL, one is still left with the difficult and opaque task of defining and optimizing a reward function. Further, because the reward function is considered as a fundamental element of the task formulation, one does not have direct access to the underlying calculations inside the reward function when considering the probabilistic modeling and inference procedure. 164 For instance, in Chapter 6, we adapted iterative amortized inference (Chapter 3) for policy optimization. In Chapter 3, in addition to gradient encoding models, which use ∇L, we considered Gaussian priors and conditional likelihoods, allowing us to define inference models that encode (prediction) errors, 𝜀x and 𝜀 z . However, in Chapter 6, because we were dealing with generic reward functions, we could only consider gradient encoding models. While similar squared error terms appear within the reward functions of the MuJoCo benchmark environments, these were effectively hidden from us. By more explicitly considering the origin and calculation of reward within the probabilistic modeling perspective, we may be able to devise more general principles for specifying and performing tasks. This thesis has focused, in part, on revitalizing ideas developed under cybernetics using modern machine learning techniques. With its focus on both biological and non-biological systems, cybernetics framed objectives in terms of homeostasis (Cannon, 1929), or the maintenance of equilibria or “setpoints” on states (Ashby, 1952). Rather than a reward function, an agent is given a desired setpoint or goal state (e.g., a thermostat temperature), or more generally, a desired probability distribution over state variables. Indeed, this perspective is still prevalent in the control literature, where many reward and cost functions are defined in terms of squared errors, i.e., Gaussian log-likelihoods, from setpoints. We can even further generalize this notion to allostasis (Sterling, 1988), yielding desired distributions over state trajectories. Thus, while Sutton and Barto, 2018 describe reward maximizing agents as “qualitatively different from equilibrium-seeking systems,” given the proper probabilistic framing, as discussed below, this distinction disappears. Defining tasks first in terms of state or trajectory distributions, rather than (simply) in terms of reward functions, could prove useful in building and understanding autonomous systems, potentially leading to improved methods for specifying and solving tasks. Task Specification Distribution-defined tasks have the benefit of naturally lending themselves toward probabilistic modeling and inference techniques. As discussed in Chapter 1, the various descendent fields of cybernetics each formulated control problems in terms of probabilistic inference. There is a rich line of work formulating planning (Attias, 2003; Botvinick and Toussaint, 2012; Piché et al., 2019) and control (Toussaint and Storkey, 2006; Todorov, 2008; Hoffman et al., 2009; Toussaint, 2009; Rawlik, Toussaint, and Vijayakumar, 2013) in terms of probabilistic inference. Within neuroscience, Friston’s line of work on active inference (K. Friston, Daunizeau, and 165 Kiebel, 2009) similarly formulates control as the inference of actions to maximize the (homeostatic) likelihood of states (Morville et al., 2018). While each of these formulations offers unique perspectives on the control problem, the commonality is that many of these are formulated in terms of desired state distributions, which then give rise to reward (or cost functions) as the resulting log-likelihood.2 In other words, for a desired state distribution, 𝑝 𝑑 (𝑆), the reward, 𝑟, that an agent receives in a particular state, 𝑠, is proportional to the log-likelihood of that state under the desired state distribution, 𝑟 (𝑠) ∝ log 𝑝 𝑑 (𝑆 = 𝑠). (9.5) For instance, a quadratic reward function between the current state and some setpoint, 𝑠∗ , i.e., 1 𝑟 (𝑠) = − ||𝑠 − 𝑠∗ ||22 , 2 is equivalent to defining a Gaussian desired state distribution with mean 𝑠∗ : 𝑝 𝑑 (𝑆) = N (𝑆; 𝑠∗ , 1), (9.6) as shown in Figure 9.3. State distributions, rather than reward functions, may be a more natural way to specify tasks. This could range from simple distributions, as in the Gaussian case, to sharp distributions, such as a Dirac delta function for a particular state, e.g., a solved Rubik’s cube. As discussed above, state distributions can also be time-varying, i.e., allostatic, defined over state trajectories. This is essential in situations where homeostatic variables need to change, e.g., blood pressure, to meet environmental requirements (Sterling, 2012). As a final note here, it is worth mentioning that the notion of “reward” itself is an abstraction, a remnant of an earlier behaviorist paradigm. Although reward, as a concept, has served as a useful tool for making progress in computational approaches to control, we should not endow it with special status. At a more fundamental level, the common characteristic of organisms, machines, and “intelligent” systems broadly is their ability to perform work to alter the state of the environment (Levin et al., 2011). Defining such systems in terms of distributions over states, rather than an inferred scalar quantity (reward), perhaps offers a more congruent framing. 2 Note that other formulations cast reward as the change in homeostatic log-likelihood (Keramati and Gutkin, 2014). 166 r(S) = log pd(S) pd(S) 0.4 0.2 0.0 −2 0 −2 −4 2 State, S −2 (a) 0 2 State, S (b) Figure 9.3: State Distributions & Rewards. A quadratic reward function, 𝑟 (𝑆), can be equivalently expressed as a Gaussian desired state distribution, 𝑝 𝑑 (𝑆). Task Solving The exponentiation interpretation above is highly similar to the control-as-inference perspective (Cooper, 1988), where we defined the likelihood of optimality as 𝑝(O = 1) ∝ exp(𝑟 (𝑠)). However, unlike control-as-inference, where the variable of interest is the abstractlydefined “optimality,” the state distribution approach considers states directly, yielding a more interpretable problem. The policy optimization objective then becomes the following cross-entropy: E𝑎∼𝜋( 𝐴) E𝑠∼𝑝env (𝑆| 𝐴=𝑎) [log 𝑝 𝑑 (𝑆 = 𝑠)] , (9.7) where, here, we have considered a single time step for simplicity. Unlike logoptimality, the objective in Eq. 9.7 has a simple interpretation: find a control policy that puts the environment in states with high log-probability under the desired distribution. In other words, make the environment state look more like your desires. Further, if we extend Eq. 9.7 to a KL divergence (Lee et al., 2019), this provides an additional state entropy term, encouraging maximum coverage over states, i.e., exploration and novelty-seeking. Desired state distributions have the immediate benefit of exposing the calculation of reward to the probabilistic formulation and inference technique. Considering the Gaussian example in Eq. 9.6, the task error, 𝜀 𝑠 ≡ 𝑠 − 𝑠∗ , is no longer hidden inside the black box of the reward function. Thus, through (iterative) amortization, we could potentially perform policy optimization by inverting this state error to yield an updated control policy. This is the idea behind negative feedback control techniques (Astrom and Murray, 2008), instantiated in so-called comparator circuits, a key concept from cybernetics (Wiener, 1948; Ashby, 1956). 167 Subtractor <latexit sha1_base64="1zRI2coJ8xVMkZJNoSsUS3P3KnM=">AAACinicdVHLSgMxFE3Hd31Vu3QTLIKrMlMFFTeiLlxWtFZoh5JJb9tgMhmSO2IZ+i1u9ZP8GzNtF33ohQuHc1+Hc6NECou+/1PwVlbX1jc2t4rbO7t7+6WDwxerU8OhwbXU5jViFqSIoYECJbwmBpiKJDSjt7u83nwHY4WOn3GYQKhYPxY9wRk6qlMqtxE+0KjsKY3QMI7ajDqlil/1x0GXQTAFFTKNeueg0Gl3NU8VxMgls7YV+AmGGTMouIRRsZ1aSBh/Y31oORgzBTbMxupH9MQxXdrTxmWMdMzOTmRMWTtUketUDAd2sZaTf9VaKfYuw0zESYoQ88mhXiopappbQbvCAEc5dIBxI5xWygcsN8EZNrfpOQizXFy+Zu58pEbFEzrL5DISVB/zfUz2tbswUP/Qgi8PJBZS56ruOgPdS4LFByyDl1o1OKvWHs8rN7fT52ySI3JMTklALsgNeSB10iCcDMkn+SLf3o5X866860mrV5jOlMlcePe/Yy/KlA==</latexit> Input Compensator- <latexit sha1_base64="R9a17PgENZo588l2PQ7Izz1U04c=">AAACg3icdVHLSgMxFE3Hd33r0k2wCIJQZmpBN0LRje4U+hDasWTS2zaYTIbkjrQM/Q+3+lf+jZnahW31QuBw7uvk3CiRwqLvfxW8ldW19Y3NreL2zu7e/sHhUdPq1HBocC21eY6YBSliaKBACc+JAaYiCa3o9S7Pt97AWKHjOo4TCBUbxKIvOENHvXQQRmhU9hAnKU66ByW/7E+DLoNgBkpkFo/dw0K309M8VRAjl8zaduAnGGbMoOASJsVOaiFh/JUNoO1gzBTYMJvKntAzx/RoXxv3YqRT9ndHxpS1YxW5SsVwaBdzOflXrp1i/zrMRP4niPnPon4qKWqae0B7wgBHOXaAcSOcVsqHzDCOzqm5SfUgzHJx+Zi59ZGaFM/obyaXkaAazdcxOdBuw1D9Qwu+3JBYSJ2ruucMdCcJFg+wDJqVcnBZrjxVS7Xb2XE2yQk5JeckIFekRu7JI2kQTgx5Jx/k01vzLryKV/0p9QqznmMyF97NN6PsyBg=</latexit> <latexit sha1_base64="cjZj5yBb3dEOBUWMCWjZNPZMCrg=">AAACjHicdVFdSxtBFJ1sW03jR2OLT30ZDIIvDbtWUBAhGCg+ppCokCxhdnKTDM4XM3clYcmP6Wv9Rf4bZ2MejGkvXDic+3U4N7NSeIzj50r04eOnre3q59rO7t7+l/rB11tvcsehx4007j5jHqTQ0EOBEu6tA6YyCXfZQ7us3z2C88LoLs4tpIpNtBgLzjBQw/rhAGGGThVtoyxoz9C4H4thvRE342XQTZCsQIOsojM8qAwHI8NzBRq5ZN73k9hiWjCHgktY1Aa5B8v4A5tAP0DNFPi0WOpf0OPAjOjYuJAa6ZJ9O1Ew5f1cZaFTMZz697WS/Fetn+P4Ii2EtjmC5q+HxrmkaGhpBh0JBxzlPADGnQhaKZ8yxzgGy9Y2dZO0KMWVa9bOZ2pRO6ZvmVKGRTVb72NyYsKFqfoPLfjmgPWQB1fNKBgYXpK8f8AmuD1tJj+bp7/PGq3r1XOq5Ds5IickIeekRW5Ih/QIJwX5Q/6Sp2g/Oosuo6vX1qiymvlG1iL69QLmWMs3</latexit> Output <latexit sha1_base64="9rBOLMnVWqHM8vfAE3N4FCbRuiY=">AAAChnicdVHLTgIxFC3jC/EB6NJNIzFxRWZ8RJdEN+7ERNQEJqRTLtDYTiftrZFM+BK3+lH+jR1kIaI3aXJy7uv03CSTwmIYfpaCldW19Y3yZmVre2e3WqvvPVjtDIcO11Kbp4RZkCKFDgqU8JQZYCqR8Jg8Xxf5xxcwVuj0HicZxIqNUjEUnKGn+rVqD+EVjcpvHWYOp/1aI2yGs6DLIJqDBplHu18v9XsDzZ2CFLlk1najMMM4ZwYFlzCt9JyFjPFnNoKuhylTYON8pnxKjzwzoENt/EuRztifHTlT1k5U4isVw7H9nSvIv3Jdh8PLOBep/xOk/HvR0EmKmhY20IEwwFFOPGDcCK+V8jEzjKM3a2HSfRTnhbhizML6RE0rR/QnU8jIUL0u1jE50n7DWP1DC77ckFlw3lU98Ab6k0S/D7AMHk6a0Wnz5O6s0bqaH6dMDsghOSYRuSAtckPapEM4ceSNvJOPoBw0g/Pg4rs0KM179slCBK0vY9jI1A==</latexit> Effector Error <latexit sha1_base64="x7aYE4NDvCzjgwSQarM3G+JqnOc=">AAACiHicdVHLSgMxFE3Hd321unQTLIKrMqNCdSeK4FLBPqAdSia90waTyZDcUcvQT3Gr3+TfmKld2FYvBA7nvk7OjVIpLPr+V8lbWV1b39jcKm/v7O7tV6oHLaszw6HJtdSmEzELUiTQRIESOqkBpiIJ7ej5tsi3X8BYoZMnHKcQKjZMRCw4Q0f1K9Uewhsald/FMXDUZtKv1Py6Pw26DIIZqJFZPPSrpX5voHmmIEEumbXdwE8xzJlBwSVMyr3MQsr4MxtC18GEKbBhPtU+oSeOGdBYG/cSpFP2d0fOlLVjFblKxXBkF3MF+Veum2F8GeYiSTOEhP8sijNJUdPCCDoQxn1Yjh1g3AinlfIRM4yjs2tu0lMQ5oW4Yszc+khNyif0N1PISFG9zdcxOdRuw0j9Qwu+3JBayJyreuAMdCcJFg+wDFpn9eC8fvZ4Ubu+mR1nkxyRY3JKAtIg1+SePJAm4eSVvJMP8umVPd9reFc/pV5p1nNI5sK7+QYAOMmF</latexit> <latexit sha1_base64="Wa83uTwDzytfXJMr3cZFJfZ0kug=">AAACg3icdVHLSgMxFE3Hd33r0k2wCIJQZmpBN0JRBJcKfQjtWDLpbRtMJkNyR1qG/odb/Sv/xkztwrZ64cLh3Nfh3CiRwqLvfxW8ldW19Y3NreL2zu7e/sHhUdPq1HBocC21eY6YBSliaKBACc+JAaYiCa3o9S6vt97AWKHjOo4TCBUbxKIvOENHvXQQRmhUdm+MNpPuQckv+9OgyyCYgRKZxWP3sNDt9DRPFcTIJbO2HfgJhhkzKLiESbGTWkgYf2UDaDsYMwU2zKayJ/TMMT3a18ZljHTK/p7ImLJ2rCLXqRgO7WItJ/+qtVPsX4eZiJMUIeY/h/qppKhp7gHtCQMc5dgBxo1wWikfMsM4OqfmNtWDMMvF5WvmzkdqUjyjv5lcRoJqNN/H5EC7C0P1Dy348kBiIXWu6p4z0L0kWHzAMmhWysFlufJULdVuZ8/ZJCfklJyTgFyRGnkgj6RBODHknXyQT2/Nu/AqXvWn1SvMZo7JXHg335d0yBI=</latexit> Feedback Takeoff <latexit sha1_base64="Gm6/O48VljMTj7pi0hOM4GOGzkE=">AAACiHicdVHLSgMxFE3Hd321unQTLIKrMqNCdScK4lLBPqAdSiZz2waTyZDcUcvQT3Gr3+TfmKld2FYvXDic+zqcG6VSWPT9r5K3srq2vrG5Vd7e2d3br1QPWlZnhkOTa6lNJ2IWpEigiQIldFIDTEUS2tHzbVFvv4CxQidPOE4hVGyYiIHgDB3Vr1R7CG9oVH4HEEeMP0/6lZpf96dBl0EwAzUyi4d+tdTvxZpnChLkklnbDfwUw5wZFFzCpNzLLKRuNRtC18GEKbBhPtU+oSeOielAG5cJ0in7eyJnytqxilynYjiyi7WC/KvWzXBwGeYiSTOEhP8cGmSSoqaFETQWBjjKsQOMG+G0Uj5ihnF0ds1tegrCvBBXrJk7H6lJ+YT+ZgoZKaq3+T4mh9pdGKl/aMGXB1ILmXNVx85A95Jg8QHLoHVWD87rZ48Xteub2XM2yRE5JqckIA1yTe7JA2kSTl7JO/kgn17Z872Gd/XT6pVmM4dkLrybb6rhyVw=</latexit> <latexit sha1_base64="lycR2ycm7YXkuP2RGdyJopVZB2A=">AAACh3icdVHJTsMwEHXDXrYCRy4WFRKnkgACjiwXjiB1k9qoctxJa9WOI3uCqKL+CVf4J/4Gp/RAWxhppKc329ObKJXCou9/lbyV1bX1jc2t8vbO7t5+5eCwaXVmODS4ltq0I2ZBigQaKFBCOzXAVCShFY0ei3rrFYwVOqnjOIVQsUEiYsEZOqpXqXQR3tCovM5GoON40qtU/Zo/DboMghmoklk89w5KvW5f80xBglwyazuBn2KYM4OCS5iUu5mFlPERG0DHwYQpsGE+lT6hp47p01gblwnSKft7ImfK2rGKXKdiOLSLtYL8q9bJML4Nc5GkGULCfw7FmaSoaeED7QsDHOXYAcaNcFopHzLDODq35jbVgzAvxBVr5s5HalI+pb+ZQkaK6m2+j8mBdheG6h9a8OWB1ELmXNV9Z6B7SbD4gGXQvKgFl7WLl6vq3cPsOZvkmJyQMxKQG3JHnsgzaRBOXsk7+SCf3pZ37l17tz+tXmk2c0Tmwrv/BvBpyQ0=</latexit> Figure 9.4: Comparator Circuit. Using the error between the current state and the input reference signal (or setpoint), a compensator-effector function updates the output control. The feedback takeoff function converts this control output into the current state. An ideal compensator-effector minimizes the discrepancy between the current state and the reference signal. Adapted from Wiener, 1948. Comparator circuits (Figure 9.4) consist of an input reference signal (or setpoint), a compensator-effector function, a feedback takeoff function, and a control output. The current state, given by the feedback takeoff function, is compared with the reference signal. The error is then fed to the compensator-effector function, which adjusts the control output to minimize this error (negative feedback). Broadly, we can identify the input reference signal as the mean of a Gaussian desired state distribution (𝑠∗ ), the compensator-effector function as a type of iterative inference model, and the feedback takeoff function as the environment (or model). While the basic comparator circuit neglects many of the underlying mathematical details and assumptions, it nevertheless provides a useful example of feedback control that appears to fit nicely with iterative amortization, and importantly, desired state distributions, where an explicit error can be calculated. Biological organisms contain negative feedback processes operating at multiple scales, altering the internal state of the organism and the external environment. From an evolutionary point of view, such homeostatic feedback control processes are only useful insofar as they enable genetic propagation (Dawkins, 1976). Thus, the homeostatic desired state distribution is itself learned through evolution, translating genetic survival into distributions over internal states (or trajectories). From this perspective, nervous systems are merely an extension of this trend, enabling more flexible forms of feedback over a range of time horizons. As argued by Damasio, 2019, this requires some method for estimating the (log) likelihood of current and future states under the homeostatic distribution, which he identifies as feelings. 168 Hierarchical Control The setup discussed thus far has only considered a single desired state distribution, recapitulating the standard RL setup from an alternative mathematical perspective. However, by framing tasks in terms of desired distributions, we can readily extend this formulation to hierarchical setups, in which higher levels assign desired perceptual distributions to lower levels (i.e., perceptual goals). These lower levels then assign desired perceptual distributions to even lower levels (or actions), attempting to maximize the likelihood of perceptions under the distribution from above. In this way, each level operates semi-autonomously as an “agent,” performing tasks in perceptual state-spaces that are increasingly coarse-grained over space and time. These perceptual spaces could, in principle, be defined in terms of generative mappings, as in Part II. Under this scheme, negative feedback and feedforward perception and control processes would operate in tandem at each level, updating perceptual estimates and goals, respectively. The brain structures associated with homeostatic regulation are evolutionarily ancient, located in the brain stem. More recent structures, such as the limbic system and cortex, provide additional layers of regulation, estimating and controlling more abstract external states. The pinnacle of this abstraction is located in prefrontal cortex, which is hypothesized to infer a hierarchy of increasingly abstract high-value (goal) states (Pezzulo, Rigoli, and K. J. Friston, 2018). Through a descending hierarchy of proprioceptive “predictions,” these goal states are translated into muscle tensions, enacted in the spinal cord through (negative feedback) reflex arcs (Adams, Shipp, and K. J. Friston, 2013). This cognitive control setup broadly agrees with perceptual control theory (Powers, 1973), which posits that actions are engaged in controlling a hierarchy of perceptual estimates via negative feedback, with homeostatic variables dictating the top of the hierarchy. This overall architecture has also been described by Meystel and Albus, 1997 from a more control-theoretic perspective. Although ideas on the relationship between feedback and the brain have existed since the dawn of cybernetics (Rosenblueth, Wiener, and Bigelow, 1943; Ashby, 1952; Powers, 1973), it is only through recent discoveries in neuroscience that a more detailed picture of higher levels, outlined above, has started to emerge (Cools, 1985; Yin, 2014; Pezzulo and Cisek, 2016; Badre, 2020). Through developments in machine learning, in part advanced through this thesis, implementation of these ideas may be on the horizon. However, while there has been progress in goal-based reinforcement learning (Schaul et al., 2015; Andrychowicz et al., 169 2017; Eysenbach, Salakhutdinov, and Levine, 2019) and goal hierarchies (Nachum et al., 2018a; Nachum et al., 2018b), substantial insights are still needed. For instance, temporal abstraction (Sutton, Precup, and Singh, 1999), despite its agreed upon importance, remains computationally elusive. Inspiration from neuroscience, particularly regarding perceptual control, as opposed to the conventional dichotomy of model-free and model-based control, may provide useful directions for progress. We have purposefully left the formulation in this section somewhat vague, as many details are left to be worked out. Yet, we feel strongly that this perspective, starting from homeostasis and scaling up toward hierarchical perceptual control, is the most promising path toward building capable systems and understanding the brain. 9.4 Bridging Neuroscience and Machine Learning The approach taken in this thesis, applying ideas from neuroscience to machine learning (and vice versa), is, unfortunately, fairly uncommon. If nothing else, we hope that the reader has come to appreciate the connections between these fields, first emphasized by cybernetics. The contributions within this thesis were inspired by ideas from theoretical neuroscience, which, in turn, originated from information theory, control theory, and cybernetics more broadly. Looking at the history of these ideas, it becomes clear that there is a proven track record of making progress through interaction between neuroscience and machine learning. Bridging these fields is difficult, but it is a well-documented path toward further progress. Indeed, there are multiple bridges between neuroscience and machine learning, each operating at different scales or in different brain regions. In this thesis, we have explored a circuit-level description for inspiration and correspondences, primarily drawn from existing work in predictive coding (Srinivasan, Laughlin, and Dubs, 1982; Rao and Ballard, 1999). These circuits are far more complex than generic neural network architectures, involving multiple objective terms, multiple functions, and specific probabilistic computations. Yet, these models are entirely abstracted away from ion currents, actions potentials, and neuromodulators. In our view, this is a useful level of abstraction for describing the computational principles underlying neural systems. Some previous works have attempted to compare similar learned models with neural data (Lotter, Kreiman, and Cox, 2018), however, additional work is needed. By scaling up these models, our hope is that future researchers can explore and test aspects of predictive coding in detail, alongside neuroscience experiments. To conclude, we will briefly review the neuroscience implications of the ideas 170 presented in this thesis. Building on the cybernetic tradition, we have investigated improved approaches to feedback and feedforward perception and control. While the structure (or circuit architecture) of these approaches is predefined, the processes themselves are learned from data. That is, we learn feedforward predictions via autoregressive flows, and we learn negative feedback updates via iterative amortization. This demonstrates that a limited set of computational operations, combined appropriately, is capable of adapting to environment-specific perception and control tasks. In a similar way, predefined neural circuit architectures, particularly in cortex, are capable of adapting to the specific perceptual and control tasks encountered by an organism during its lifetime. Notably, in addition to the basic arithmetic operations for probabilistic computations, e.g., subtraction for error and multiplication for precision-weighting, these approaches rely on learned non-linear mappings, parameterized by deep networks. By identifying the probabilistic computations as occurring across neurons, as in predictive coding, the non-linear mappings are implicated in occurring within neurons. Importantly, this interpretation hinges on implementing negative feedback (inference) optimization using amortization. Thus, a significant portion of neurons may be devoted to learning to optimize the perceptual and control estimates of other neural circuits. While this idea has a long history (Rosenblueth, Wiener, and Bigelow, 1943), amortization provides a new lens for understanding how such corrective processes can be learned. Much work remains to be done in formulating and understanding neural systems, and biological systems generally, in computational terms. Humans, in particular, through a complex array of perceptual and control processes, are capable of bringing about wildly improbable, high-value environmental states. While it is tempting to think that we are close to “solving intelligence,” we are still only scratching the surface of basic engineering principles. Substantial work remains to be done in integrating and scaling these systems. May we continue to look toward the complexity and wonder of biological systems as a guiding source of inspiration. References Adams, Rick A, Stewart Shipp, and Karl J Friston (2013). “Predictions not commands: active inference in the motor system”. In: Brain Structure and Function 218.3, pp. 611–643. Albus, James, Anthony J Barbera, and Roger N Nagel (1980). Theory and practice of hierarchical control. National Bureau of Standards. 171 Andrychowicz, Marcin et al. (2017). “Hindsight experience replay”. In: Advances in neural information processing systems, pp. 5048–5058. Ashby, W Ross (1952). Design for a brain: The origin of adaptive behaviour. – (1956). An Introduction to Cybernetics. Chapman and Hall. Astrom, Karl Johan and Richard M Murray (2008). Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press. Attias, Hagai (2003). “Planning by probabilistic inference.” In: AISTATS. Citeseer. Badre, David (2008). “Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes”. In: Trends in cognitive sciences 12.5, pp. 193–200. – (2020). “On Task”. In: On Task. Princeton University Press. Botvinick, Matthew and Marc Toussaint (2012). “Planning as inference”. In: Trends in cognitive sciences 16.10, pp. 485–488. Cannon, Walter B (1929). “Organization for physiological homeostasis”. In: Physiological reviews 9.3, pp. 399–431. Cools, AR (1985). “Brain and behavior: hierarchy of feedback systems and control of input”. In: Perspectives in ethology. Springer, pp. 109–168. Cooper, Gregory F (1988). “A method for using belief networks as influence diagrams”. In: Fourth Workshop on Uncertainty in Artificial Intelligence. Damasio, Antonio (2019). The strange order of things: Life, feeling, and the making of cultures. Vintage. Dawkins, Richard (1976). The selfish gene. Oxford university press. Dayan, Peter and Geoffrey Hinton (1993). “Feudal reinforcement learning”. In: Advances in Neural Information Processing Systems. Eysenbach, Ben, Russ R Salakhutdinov, and Sergey Levine (2019). “Search on the replay buffer: Bridging planning and reinforcement learning”. In: Advances in Neural Information Processing Systems, pp. 15246–15257. Friston, Karl, Jean Daunizeau, and Stefan J Kiebel (2009). “Reinforcement learning or active inference?” In: PloS one 4.7, e6421. Greff, Klaus et al. (2019). “Multi-Object Representation Learning with Iterative Variational Inference”. In: International Conference on Machine Learning, pp. 2424– 2433. Ha, David and Jürgen Schmidhuber (2018). “Recurrent world models facilitate policy evolution”. In: Advances in Neural Information Processing Systems, pp. 2450– 2462. Hafner, Danijar et al. (2019). “Learning Latent Dynamics for Planning from Pixels”. In: International Conference on Machine Learning, pp. 2555–2565. 172 Hoffman, Matthew et al. (2009). “An expectation maximization algorithm for continuous Markov decision processes with arbitrary reward”. In: Artificial Intelligence and Statistics, pp. 232–239. Keramati, Mehdi and Boris Gutkin (2014). “Homeostatic reinforcement learning for integrating reward collection and physiological stability”. In: Elife 3, e04811. Lee, Lisa et al. (2019). “Efficient exploration via state marginal matching”. In: arXiv preprint arXiv:1906.05274. Levin, Robert et al. (2011). Work meets life: exploring the integrative study of work in living systems. MIT Press. Levine, Sergey (2018). “Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review”. In: arXiv preprint arXiv:1805.00909. Lotter, William, Gabriel Kreiman, and David Cox (2018). “A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception”. In: arXiv preprint arXiv:1805.10734. Marino, Joseph and Yisong Yue (2019). “An Inference Perspective on Model-Based Reinforcement Learning”. In: ICML Workshop on Generative Modeling and Model-Based Reasoning for Robotics and AI. url: https://drive.google. com/open?id=1uc1FdjHVkkEAaU6jz7aVrP3khGyqkdKi. Meystel, Al M and James Albus (1997). “Intelligent Systems”. In: Architecture, Designand Control. Morville, Tobias et al. (2018). “The homeostatic logic of reward”. In: bioRxiv, p. 242974. Nachum, Ofir et al. (2018a). “Data-efficient hierarchical reinforcement learning”. In: Advances in Neural Information Processing Systems, pp. 3307–3317. – (2018b). “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning”. In: International Conference on Learning Representations. Pezzulo, Giovanni and Paul Cisek (2016). “Navigating the affordance landscape: feedback control as a process model of behavior and cognition”. In: Trends in cognitive sciences 20.6, pp. 414–424. Pezzulo, Giovanni, Francesco Rigoli, and Karl J Friston (2018). “Hierarchical active inference: A theory of motivated control”. In: Trends in cognitive sciences 22.4, pp. 294–306. Piché, Alexandre et al. (2019). “Probabilistic Planning with Sequential Monte Carlo methods”. In: International Conference on Learning Representations. url: https://openreview.net/forum?id=ByetGn0cYX. Powers, William T (1973). Behavior: The control of perception. Aldine Chicago. Rao, Rajesh PN and Dana H Ballard (1999). “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” In: Nature neuroscience 2.1. 173 Rawlik, Konrad, Marc Toussaint, and Sethu Vijayakumar (2013). “On stochastic optimal control and reinforcement learning by approximate inference”. In: TwentyThird International Joint Conference on Artificial Intelligence. Rosenblueth, Arturo, Norbert Wiener, and Julian Bigelow (1943). “Behavior, purpose and teleology”. In: Philosophy of science 10.1, pp. 18–24. Schaul, Tom et al. (2015). “Universal value function approximators”. In: International conference on machine learning, pp. 1312–1320. Srinivasan, Mandyam Veerambudi, Simon Laughlin, and Andreas Dubs (1982). “Predictive coding: a fresh view of inhibition in the retina”. In: Proceedings of the Royal Society of London. Series B. Biological Sciences 216.1205, pp. 427–459. Sterling, Peter (1988). “Allostasis: a new paradigm to explain arousal pathology”. In: Handbook of life stress, cognition and health. – (2012). “Allostasis: a model of predictive regulation”. In: Physiology & behavior 106.1, pp. 5–15. Sutton, Richard S and Andrew G Barto (2018). Reinforcement learning: An introduction. MIT press. Sutton, Richard S, Doina Precup, and Satinder Singh (1999). “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning”. In: Artificial intelligence 112.1-2, pp. 181–211. Todorov, Emanuel (2008). “General duality between optimal control and estimation”. In: Decision and Control, 2008. CDC 2008. 47th IEEE Conference on. IEEE, pp. 4286–4292. Toussaint, Marc (2009). “Robot trajectory optimization using approximate inference”. In: International conference on machine learning, pp. 1049–1056. Toussaint, Marc and Amos Storkey (2006). “Probabilistic inference for solving discrete and continuous state Markov Decision Processes”. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp. 945–952. Vezhnevets, Alexander Sasha et al. (2017). “Feudal networks for hierarchical reinforcement learning”. In: International Conference on Machine Learning. PMLR, pp. 3540–3549. Wiener, Norbert (1948). Cybernetics or Control and Communication in the Animal and the Machine. MIT press. Wolpert, Daniel M, Zoubin Ghahramani, and J Randall Flanagan (2001). “Perspectives and problems in motor learning”. In: Trends in cognitive sciences 5.11, pp. 487– 494. Yin, Henry H (2014). “How basal ganglia outputs generate behavior”. In: Advances in neuroscience. 174 Yin, Henry H (2016). “The basal ganglia and hierarchical control in voluntary behavior”. In: The Basal Ganglia. Springer, pp. 513–566.