HIPPOCAMPAL REPRESENTATIONS EMERGE WHEN TRAINING RECURRENT NEURAL NETWORKS ON A MEMORY DEPENDENT MAZE NAVIGATION TASK

Abstract

Can neural networks learn goal-directed behaviour using similar strategies to the brain, by combining the relationships between the current state of the organism and the consequences of future actions? Recent work has shown that recurrent neural networks trained on goal based tasks can develop representations resembling those found in the brain, entorhinal cortex grid cells, for instance. Here we explore the evolution of the dynamics of their internal representations and compare this with experimental data. We observe that once a recurrent network is trained to learn the structure of its environment solely based on sensory prediction, an attractor based landscape forms in the network's representation, which parallels hippocampal place cells in structure and function. Next, we extend the predictive objective to include Q-learning for a reward task, where rewarding actions are dependent on delayed cue modulation. Mirroring experimental findings in hippocampus recordings in rodents performing the same task, this training paradigm causes nonlocal neural activity to sweep forward in space at decision points, anticipating the future path to a rewarded location. Moreover, prevalent choice and cueselective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that combining predictive, unsupervised learning of the structure of an environment with reinforcement learning can help understand the formation of hippocampus-like representations containing both spatial and task-relevant information.

1. INTRODUCTION

Recurrent neural networks have been used to perform spatial navigation tasks and the subsequent study of their internal representations has yielded dynamics and structures that are strikingly biological. Metric (Cueva & Wei, 2018; Banino et al., 2018) and non-metric (Recanatesi et al., 2019) representations mimicking grid (Fyhn et al., 2004) and place cells (O' Keefe & Nadel, 1978) respectively form once the recurrent network has learned a predictive task in the context of a complex environment. Cueva et al. (2020) demonstrates not only the emergence of characteristic neural representations, but also hallmarks of head direction system cells when training a recurrent network on a simple angular velocity integration task. Biologically, non-metric representations are associated with landmark spatial memory, in which place cells within the mammalian hippocampus fire when the associated organism is present in a corresponding place field. Extrafield firing of place cells occurs when these neurons spike outside of these contiguous place field regions. Here we show that recurrent neural networks (RNNs) not only form corresponding attractor landscapes, but also produce representations with internal dynamics that closely resemble those found experimentally in the hippocampus when performing goal-directed behaviour. Research in neuroscience such as that of Johnson & Redish (2007) , shows that spatial representations in mice in the CA3 region of the hippocampus frequently fire nonlocally. Griffin et al. (2007) show that a far higher proportion of hippocampal neurons in the CA1 region in rats performing an episodic task in a T-shaped maze encode the phase of the task rather than spatial information (in this case trajectory direction). Ainge et al. (2007) show CA1 place cells encode destination location at the start position of a maze. Lee et al. (2006) demonstrate that place fields of CA1 neurons gradually drift toward reward locations throughout reward training on a T-shaped maze.

