Three-factor rules: from reward to novelty and surprise

Abstract: 

A large fraction of behavior and cognitive neuroscience have been influenced by reinforcement learning theory. However, behavior also happens in the absence of reward. I will explain how concepts of reinforcement learning can be translated to situations and how novelty and surprise can act as internal brain signals, similar to, but distinct from the TD-error signals used in reinforcement learning [1-4]

[1] W. Gerstner, M. Lehmann, V. Liakoni, and J. Brea (2018) 
Eligibility traces and plasticity on behavioral time scales: experimental support of NeoHebbian three-factor learning rules. 
Front. Neural Circuits, 12:53 doi: 10.3389/fncir.2018.00053

[2] A. Modirshanechi, J. Brea, and W. Gerstner (2022) 
A taxonomy of surprise definitions 
J. Mathem. Psychol. 110:102712, DOI: 10.1016/j.jmp.2022.102712 

[3] A. Modirshanechi, W.-H. Lin, H.A. Xu, M. Herzog, and W. Gerstner (2025) 
Novelty as a drive of human exploration in complex stochastic environments 
Proc. Natl. Acad. Sci. (U.S.A), 122:e2502193122 DOI 10.1073/pnas.2502193122

[4] H.A. Xu, A. Modirshanechi, M.P. Lehmann, W. Gerstner, M.H. Herzog (2021) 
Novelty is not Surprise: Human exploratory and adaptive behavior in sequential decision-making 
PLoS Comput Biol 17: e1009070. doi: https://doi.org/10.1371/journal.pcbi.1009070 

[5] V. Liakoni, A. Modirshanechi, W. Gerstner, and J. Brea (2021) 
Learning in Volatile environments with the Bayes Factor Surprise 
Neural Computation 33: 1-72