A reinforcement learning account of planning, prospective simulation, and hippocampal replay

To make decisions, we must evaluate candidate choices by accessing records of relevant experiences. Yet little is known about which experiences the brain considers or ignore during planning, which ultimately affects choice. In this talk, I will describe my research revealing principles by which we use our memories to plan and decide. First, I will describe a normative theory predicting which memories would be ideally accessed at each moment to optimize future decisions. Using nonlocal “replay” of spatial locations in hippocampus as a window into memory access, I will show simulations of a spatial navigation task where an ideal agent accesses memories of locations sequentially, ordered by utility: how much extra reward would be earned due to better choices. This prioritization balances two desiderata: the need to evaluate imminent choices, vs. the gain from propagating newly encountered information to preceding locations. In addition to explaining the role of memory in planning, this theory offers a simple explanation for numerous findings about place cells and unifies seemingly disparate proposed functions of replay including planning, learning, and consolidation. I will then present an experimental framework using neuroimaging in humans to predict and measure memory reactivation during planning and its effect on choice, including techniques from machine learning and network science. Finally, I will describe a broader research program for understanding the neural mechanisms of how we plan and decide and the implications for related psychiatric disorders such as rumination and craving.
 
Bio
Marcelo Mattar is a Newton International Postdoctoral Fellow working at University of Cambridge and Princeton University with Máté Lengyel and Nathaniel Daw. He studies learning and decision-making using a combination of theoretical and human behavioral/imaging approaches, with a particular interest in reinforcement learning and Bayesian inference. He completed his PhD in Psychology at the University of Pennsylvania, where he studied network theory with Danielle Bassett and visual adaptation with Geoffrey Aguirre.

Video transmission using zoom : https://epfl.zoom.us/j/9946495775