Back to the future: episodic memories for control
Máté Lengyel1 and Peter Dayan2
1Collegium Budapest Institute for Advanced Study, 2Gatsby Computational Neuroscience Unit, UCL

Episodic (or episodic-like) memories have traditionally been discussed in terms of remembering specific events in one's personal past [1]. However, there is an increasing realisation that remembering the past is mainly important for planning for the future [2]. Indeed, for example, experimental evidence shows that amnesic patients have difficulties with imagining future events [3]. However, from a computational standpoint, semantic, rather than episodic, memories seem to be a much better substrate of future planning: why should it be better to act on the basis of the apparently statistically inefficient recollection of single happenings (episodic, memory-based control), rather than the seemingly normative use of accumulated experience from multiple events (semantic, model-based control)?

We suggest that the key issue is computational noise. Particularly in interestingly complex environments, semantic control, which depends on noisy recursive operations to calculate the values of actions, can readily be bested by episodic control, which does not, particularly in the face of very limited experience. Processing noise overwhelms the statistical inefficiency of basing decisions on individual episodes rather than their integrated statistics. We show this using detailed mathematical analyses and numerical simulations in the normative framework of reinforcement learning [4]. Our analysis parallels earlier work [5], which showed that, again because of computational noise, semantic, model-based control, is less accurate than habitual control after more substantial amounts of experience.

Besides giving a normative benefit for episodic memories, our analysis also allows us to interpret experimental data on systems-level consolidation (e.g. [6]) as the hallmark of the transfer of control, rather than the transfer of memories, from the hippocampus to the neocortex, or the striatum [7]. Furthermore, in the light of these results, the fact that neural networks tend to forget past episodes in favour of recent ones may seem less catastrophic than often mooted [9].

[1] E. Tulving. Episodic and semantic memory. In E. Tulving and W. Donaldson, eds, Organisation of Memory, pp. 381-403. Academic Press, 1972.
[2] Y. Dudai and M. Carruthers. The Janus face of Mnemosyne. Nature 434:567, 2005.
[3] D. Hassabis, D. Kumaran, S.D. Vann, and E.A. Maguire. Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci U.S.A. 104:1726-1731, 2007.
[4] R.S. Sutton and A.G. Barto. Reinforcement Learning. MIT Press, 1998.
[5] N.D. Daw, Y. Niv, and P. Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8:1704-1711, 2005.
[6] D. Tse, R.F. Langston, M. Kakeyama, I. Bethus, P.A. Spooner, E.R. Wood, M.P. Witter, and R.G.M. Morris. Schemas and memory consolidation. Science 316:76-82, 2007.
[7] M.G. Packard and J.L. McGaugh. Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: further evidence for multiple memory systems. Behav. Neurosci. 106:439-446, 1992.
[8] S. Fusi and L.F. Abbott. Limits on the memory storage capacity of bounded synapses. Nat. Neurosci. 10:485-493, 2007.