New Algorithms for Programmatic Deep Learning with Applications to Behavior Modeling

Author: Zhan, Eric

Year: 2022

Degree: Dissertation (Ph.D.)

Advisor: Yue, Yisong

Committee Members: Murray, Richard M.; Perona, Pietro; Chaudhuri, Swarat; Yue, Yisong

Option: Computing and Mathematical Sciences

DOI: 10.7907/5n5q-x203

Abstract

Raw behavioral data is becoming increasingly more abundant and more easily obtainable in spatiotemporal domains such as sports, video games, navigation & driving, motion capture, and animal science. How can we best use this data to advance their respective domains forward? For instance, researchers for self-driving vehicles would like to identify the key features of the environment state that impact decision-making the most; game developers would like to populate their games with characters that have unique and diverse behaviors to create a more immersive gaming experience; and behavioral neuroscientists would like to uncover the underlying mechanisms that drive learning in animals. Machine learning, the science of developing models and algorithms to identify and leverage patterns in data, is well-equipped to aid in these endeavors. But how do we integrate machine learning with these spatiotemporal domains in a principled way? In this dissertation, we develop and introduce new algorithms in programmatic deep learning that tackle some of the new challenges encountered in behavior modeling.

Our work in programmatic deep learning comprises two main themes: in the first, we show how to use expert-written programs as sources of weak labels in domains where manually-annotated expert labels are scarce; in the second, we explore programs as a flexible function class with human-interpretable structure and show how to learn them via neurosymbolic program learning. Augmenting deep learning with programmatic structure allows domain experts to easily incorporate domain knowledge into machine learning models; we show that this results in significant improvements in many behavior modeling applications like imitation learning, controllable generation, counterfactual analysis, and unsupervised clustering.

Files