Research
Research Topics
Publications
Depression / Computational Psychiatry
PhD thesis
- pdf Reinforcers and control: Towards a computational ætiology of depression
- Gatsby Computational Neuroscience Unit, UCL (2007)
- Depression, like many psychiatric disorders, is a disorder of affect. Over the past decades, a large number of affective issues in depression have been characterised, both in human experiments and animal models of the disorder. Over the same period, experimental neuroscience, helped by computational theories such as reinforcement learning, has provided detailed descriptions of the psychology and neurobiology of affective decision making. Here, we attempt to harvest the advances in the understanding of the brain's normal dealings with rewards and punishments to dissect out and define more clearly the components that make up depression. We start by exploring changes to primary reinforcer sensitivity in the learned helplessness animal models of depression. Then, a detailed formalisation of control in a goal-directed decision making framework is presented and related to animal and human data. Finally, we show how serotonin's joint involvement in reporting negative values and inhibiting actions may explain some aspects of its involvement in depression. Throughout, aspects of depression are seen as emerging from normal affective function and reinforcement learning, and we thus conclude that computational descriptions of normal affective function provide one possible avenue by which to define an ætiology of depression.
Under Review
- Altered Pavlovian control of withdrawal is a state marker of depression
- Neural correlates of reversal learning in schizophrenia
- Optimism as a prior belief about the probability of future reward
- Pavlovian-Instrumental Transfer as learning paradigm to investigate alcohol dependence
- Stressful life events moderate the relationship between fluid intelligence and prediction error signalling
- preprint Computational Psychiatry
- Encyclopædia of Computational Neuroscience
-
Computational Psychiatry is a heterogeneous field at the intersection of computational neuroscience and psychiatry. Incorporating methods from psychiatry, psychology, neuroscience, behavioural economics and machine learning, computational psychiatry focuses on building mathematical models of neural or cognitive phenomena relevant to psychiatric diseases. The models span a wide range - from biologically detailed models of neurons or networks to abstract models describing high-level cognitive abilities of an organism. Psychiatric diseases are conceptualized as either an extreme of normal function, or as a consequence of alterations in parts of the model.
As in computational neuroscience more generally, the building of models forces key concepts to be made concrete and hidden assumptions to be made explicit. One critical functions of these models in the setting of psychiatry are their ability to bridge between low-level biological and high-level cognitive features. While many neurobiological alterations are known, the exclusively atheoretical focus of standard psychiatric nosology on high-level symptoms has as yet prevented an integration of these bodies of knowledge. David Marr pointed out that models at different levels may be independent (Marr, 1982). Nevertheless, implementational details may constrain functions at the computational level. The models used in computational psychiatry make these constraints explicit, and thereby aim to provide normative conduits between the different levels at which neural systems are analysed (Stephan et al., 2006; Huys et al., 2011; Hasler, 2012; Montague et al., 2012). This in turn allows for a principled approach to study dysfunctions, and indeed may allow the dysfunctions observed in psychiatry to inform neuroscience in general.
Practically, it underpins hopes that computational techniques may facilitate the development of a psychiatric nomenclature based on an understanding of the underlying neuroscience. Computational models enhance experimental designs by allowing more intricate neural and/or cognitive processes to be inferred from complex features of the data, often via Bayesian inference. These aspects motivate hopes that it may facilitate the development of clinical treatment decision tools informed by advances in neuroscience.
- Differential, but not opponent, effects of L-DOPA and citalopram on action learning with reward and punishment
Peer Reviewed
- preprint
- Aversive Pavlovian control of instrumental behaviour in humans
- J. Cogn. Neurosci. (2013): In press
- Adaptive behaviour involves interactions between systems regulating Pavlovian and instrumental control of actions. Here, we present the first investigation of the neural mechanisms underlying aversive Pavlovian-instrumental transfer using fMRI in humans. Recent evidence indicates that these Pavlovian influences on instrumental actions are action-specific: Instrumental approach is invigorated by appetitive Pavlovian cues, but inhibited by aversive Pavlovian cues. Conversely, instrumental withdrawal is inhibited by appetitive Pavlovian cues, but invigorated by aversive Pavlovian cues. We show that BOLD responses in the amygdala and the nucleus accumbens were associated with behavioural inhibition by aversive Pavlovian cues, irrespective of action context. Furthermore, BOLD responses in the ventromedial prefrontal cortex differed between approach and withdrawal actions. Aversive Pavlovian conditioned stimuli modulated connectivity between the ventromedial prefrontal cortex and the caudate nucleus. These results show that action-specific aversive control of instrumental behaviour involves the modulation of fronto-striatal interactions by Pavlovian conditioned stimuli.
- preprint Mapping anhedonia onto reinforcement learning. A behavioural meta-analysis.
- Biology of Mood and Anxiety Disorders (2013): In Press
-
Background: Depression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.
Methods: Six behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.
Results: MDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.
Conclusion: Reward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.
- doi pdf Frontal theta overrides Pavlovian learning biases
- J. Neurosci. (2013) 33(19):8541-8548
- Pavlovian biases influence learning and decision making by intricately coupling reward seeking with action invigoration and punishment avoidance with action suppression. This bias is not always adaptive; it can oftentimes interfere with instrumental requirements. The prefrontal cortex is thought to help resolve such conflict between motivational systems, but the nature of this control process remains unknown. EEG recordings of mid-frontal theta band power are sensitive to conflict and predictive of adaptive control over behavior, but it is not clear whether this signal would reflect control over conflict between motivational systems. Here we utilized a task that orthogonalized action requirements and outcome valence while recording concurrent EEG in human participants. By applying a computational model of task performance, we derived parameters reflective of the latent influence of Pavlovian bias and how it was modulated by mid- frontal theta power during motivational conflict. Between subjects, individuals who performed better under Pavlovian conflict exhibited higher mid-frontal theta power. Within subjects, trial- to-trial variance in theta power was predictive of ability to overcome the influence of the Pavlovian bias, and this effect was most pronounced in individuals with higher mid-frontal theta to conflict. These findings demonstrate that mid-frontal theta is not only a sensitive index of prefrontal control, but it can also reflect the application of top-down control over instrumental processes.
- pdf Learned helplessness and generalization
- Proceedings of the 35th Annual Conference of the Cognitive Science Society (2013)
- In learned helplessness experiments, subjects first experience a lack of control in one situation, and then show learning deficits when performing or learning another task in another situation. Generalization, thus, is at the core of the learned helplessness phenomenon. Substantial experimental and theoretical effort has been invested into establishing that a state- and task-independent belief about controllability is necessary. However, to what extent generalization is also sufficient to explain the transfer has not been examined. Here, we show qualitatively and quantitatively that Bayesian learning of action-outcome contingencies at three levels of abstraction is sufficient to account for the key features of learned helplessness, including escape deficits and impairment of appetitive learning after inescapable shocks.
- doi pdf Dopamine restores reward prediction errors in older age
- Nature Neuroscience 16, 648-653 (2013)
- Senescence affects the ability to utilize information about the likelihood of rewards for optimal decision-making. Using functional magnetic resonance imaging in humans, we found that healthy older adults had an abnormal signature of expected value, resulting in an incomplete reward prediction error (RPE) signal in the nucleus accumbens, a brain region that receives rich input projections from substantia nigra/ventral tegmental area (SN/VTA) dopaminergic neurons. Structural connectivity between SN/VTA and striatum, measured by diffusion tensor imaging, was tightly coupled to inter-individual differences in the expression of this expected reward value signal. The dopamine precursor levodopa (L-DOPA) increased the task-based learning rate and task performance in some older adults to the level of young adults. This drug effect was linked to restoration of a canonical neural RPE. Our results identify a neurochemical signature underlying abnormal reward processing in older adults and indicate that this can be modulated by L-DOPA.
- doi pdf Go and nogo learning in reward and punishment: Interactions between affect and effect
- Neuroimage (2012) 62(1):154-66
- Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.
- doi pdf Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees
- PLoS Comp Biol (2012) 8(3): e1002410
- When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal- directed choices in a manner that may be important for theories of decision-making in mood disorders.
- doi pdf Ventral striatal prediction error signalling is associated with dopamine synthesis capacity and fluid intelligence
- Human Brain Mapping (2012)
- Fluid intelligence represents the capacity for flexible problem solving and rapid behavioral adaptation. Rewards drive flexible behavioral adaptation, in part via a teaching signal expressed as reward prediction errors in the ventral striatum, which has been associated with phasic dopamine release in animal studies. We examined a sample of 28 healthy male adults using multimodal imaging and biological parametric mapping with (1) functional magnetic resonance imaging during a reversal learning task and (2) in a subsample of 17 subjects also with positron emission tomography using 6-[(18) F]fluoro-L-DOPA to assess dopamine synthesis capacity. Fluid intelligence was measured using a battery of nine standard neuropsychological tests. Ventral striatal BOLD correlates of reward prediction errors were positively correlated with fluid intelligence and, in the right ventral striatum, also inversely correlated with dopamine synthesis capacity (FDOPA K#inapp). When exploring aspects of fluid intelligence, we observed that prediction error signaling correlates with complex attention and reasoning. These findings indicate that individual differences in the capacity for flexible problem solving relate to ventral striatal activation during reward-related learning, which in turn proved to be inversely associated with ventral striatal dopamine synthesis capacity.
- doi pdf Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain
- J. Neurosci (2011) 31(21):7867-75
- The acquisition of reward and the avoidance of punishment could logically be contingent on either emitting or withholding particular actions. However, the separate pathways in the striatum for go and no-go appear to violate this independence, instead coupling affect and effect. Respect for this interdependence has biased many studies of reward and punishment, so potential action-outcome valence interactions during anticipatory phases remain unexplored. In a functional magnetic resonance imaging study with healthy human volunteers, we manipulated subjects' requirement to emit or withhold an action independent from subsequent receipt of reward or avoidance of punishment. During anticipation, in the striatum and a lateral region within the substantia nigra/ventral tegmental area (SN/VTA), action representations dominated over valence representations. Moreover, we did not observe any representation associated with different state values through accumulation of outcomes, challenging a conventional and dominant association between these areas and state value representations. In contrast, a more medial sector of the SN/VTA responded preferentially to valence, with opposite signs depending on whether action was anticipated to be emitted or withheld. This dominant influence of action requires an enriched notion of opponency between reward and punishment.
- doi pdf Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding
- PLoS Comp Biol (2011) 7(4): e1002028
- Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.
- doi pdf Are computational models useful for psychiatry?
- Neural Networks (2011) 24(6):544-551
- Mathematically rigorous descriptions of key hypotheses and theories are becoming more common in neuroscience and are beginning to be applied to psychiatry. In this article two fictional characters, Dr. Strong and Mr. Micawber, debate the use of such computational models (CMs) in psychiatry. We present four fundamental challenges to the use of CMs in psychiatry: (a) the applicability of mathematical approaches to core concepts in psychiatry such as subjective experiences, conflict and suffering; (b) whether psychiatry is mature enough to allow informative modelling; (c) whether theoretical techniques are powerful enough to approach psychiatric problems; and (d) the issue of communicating clinical concepts to theoreticians and vice versa. We argue that CMs have yet to influence psychiatric practice, but that they help psychiatric research in two fundamental ways: (a) to build better theories integrating psychiatry with neuroscience; and (b) to enforce explicit, global and efficient testing of hypotheses through more powerful analytical methods. CMs allow the complexity of a hypothesis to be rigorously weighed against the complexity of the data. The paper concludes with a discussion of the path ahead. It points to stumbling blocks, like the poor communication between theoretical and medical communities. But it also identifies areas in which the contributions of CMs will likely be pivotal, like an understanding of social influences in psychiatry, and of the co-morbidity structure of psychiatric diseases.
- pdf Computational unhappiness: Modelling depression
- Frontiers in Neurosciences (2009) 3(2):263
- doi pdf Serotonin in affective control
- Annu Rev Neurosci (2009) :: 32: 95-126
- Serotonin is a neuromodulator that is extensively entangled in fundamental aspects of brain function and behavior. We present a computational view of its involvement in the control of appetitively and aversively motivated actions. We first describe a range of its effects in invertebrates, endowing specific structurally fixed networks with plasticity at multiple spatial and temporal scales. We then consider its rather widespread distribution in the mammalian brain. We argue that this is associated with a more unified representational and functional role in aversive processing that is amenable to computational analyses with the kinds of reinforcement learning techniques that have helped elucidate dopamine's role in appetitive behavior. Finally, we suggest that it is only a partial reflection of dopamine because of essential asymmetries between the natural statistics of rewards and punishments.
- doi pdf A Bayesian formulation of behavioral control
- Cognition (2009) Special Issue
- Helplessness, a belief that the world is not subject to behavioral control, has long been central to our understanding of depression, and has influenced cognitive theories, animal models and behavioral treatments. However, despite its importance, there is no fully accepted definition of helplessness or behavioral control in psychology or psychiatry, and the formal treatments in engineering appear to capture only limited aspects of the intuitive concepts. Here, we formalize controllability in terms of characteristics of prior distributions over affectively charged environments. We explore the relevance of this notion of control to reinforcement learning methods of optimising behavior in such environments and consider how apparently maladaptive beliefs can result from normative inference processes. These results are discussed with reference to depression and animal models thereof.
- pdf Psychiatry: insights into depression through normative decision-making models
- NIPS 2008
- Decision making lies at the very heart of many psychiatric diseases. It is also a central theoretical concern in a wide variety of fields and has undergone detailed, in-depth, analyses. We take as an example Major Depressive Disorder (MDD), applying insights from a Bayesian reinforcement learning framework. We focus on anhedonia and helplessness. Helplessness-a core element in the conceptual- izations of MDD that has lead to major advances in its treatment, pharmacolog- ical and neurobiological understanding-is formalized as a simple prior over the outcome entropy of actions in uncertain environments. Anhedonia, which is an equally fundamental aspect of the disease, is related to the effective reward size. These formulations allow for the design of specific tasks to measure anhedonia and helplessness behaviorally. We show that these behavioral measures capture explicit, questionnaire-based cognitions. We also provide evidence that these tasks may allow classification of subjects into healthy and MDD groups based purely on a behavioural measure and avoiding any verbal reports.
- doi pdf suppl Serotonin, behavioral inhibition and negative mood
- PLoS Computational Biology (2008) 4(2): e4
- Pavlovian predictions of future aversive outcomes lead to behavioral inhibition, suppression, and withdrawal. There is considerable evidence for the involvement of serotonin in both the learning of these predictions and the inhibitory consequences that ensue, although less for a causal relationship between the two. In the context of a highly simplified model of chains of affectively charged thoughts, we interpret the combined effects of serotonin in terms of pruning a tree of possible decisions, (i.e., eliminating those choices that have low or negative expected outcomes). We show how a drop in behavioral inhibition, putatively resulting from an experimentally or psychiatrically influenced drop in serotonin, could result in unexpectedly large negative prediction errors and a significant aversive shift in reinforcement statistics. We suggest an interpretation of this finding that helps dissolve the apparent contradiction between the fact that inhibition of serotonin reuptake is the first-line treatment of depression, although serotonin itself is most strongly linked with aversive rather than appetitive outcomes and predictions.
Book chapters
- Neurobiological understanding and computational models of addictive behavior
- Mishara et al. (ed.) (2011) Phenomenological neuropsychiatry: bridging the clinic and clinical neuroscience
Abstracts / Unrefereed
- Habitual and goal-directed learning in alcohol dependence
- Dopamine (2013)
- Representations for planning
- Einstein Symposium Berlin (2013)
- poster Controllability and resource-rational planning
- Cosyne (2013)
- Optimism can function as a normative prior belief on the probability of rewards
- Mathematical Biosciences Institute, Workshop on Cognitive Neuroscience, Columbus Ohio, USA, December 2012
- poster The neural bases of reversal learning deficits in unmedicated schizophrenia patients
- Einstein meeting Berlin (2011)
- poster Psychopathy: a dysfunction in Pavlovian-to-instrumental transfer
- Biological Psychiatry (2011)
- doi Approaching avoidance: instrumental and Pavlovian asymmetries in the procesing of rewards and punishments
- Cosyne 2010
- doi Bonsai trees: how the Pavlovian system sculpts sequential decisions
- Cosyne 2010
- Reward and helplessness in depression
- Cosyne 2008
- Normative learning: a route to depression?
- CNS 2007
- Depression as normal learning
- Nature medicine-roche conference on translational psychiatry
- poster Optimal helplessness: a normative framework for depression
- CNS 2006
- poster Malignant evaluation: reinforcement learning, neuromodulation and depression
- Cosyne 2006
Population coding
Peer reviewed
- pdf Probabilistic computation in spiking populations
- NIPS 2004
- doi pdf code Fast population coding
- Neural Computation (2007) 19(2):460-97
- doi pdf Encoding and decoding spikes for dynamic stimuli
- Neural Computation (2008) 20(9):2325-60
Abstracts / Unrefereed
- abstract talk Fast population coding
- CNS 2006
- Dynamic population codes for sensorimotor processing
- Cosyne 2006
- poster Population coding in a fast-changing world
- Cosyne 2005
- Representational pursuit: population codes for dynamic environments
- Cosyne 2005
- poster A simple population code in a fast-changing world
- Soc. Neurosci. Abstr. 2005
Single cells
Peer reviewed
- doi pdf code movie Smoothing of, and parameter estimation from, noisy biophysical recordings
- PLoS Computational Biology (2009) 5(5): e1000379
- doi pdf code Efficient estimation of detailed single-neuron models
- J. Neurophysiol. (2006) 96: 872-890
- pdf Large-scale biophysical parameter estimation in single neurons via constrained linear regression
- NIPS 2005
Abstracts / Unrefereed
- poster Estimating non-homogeneous channel densities and synaptic activity from spatiotemporal dendritic voltage recordings
- Cosyne 2005
- poster Estimating non-homogeneous channel densities and synaptic activity from spatiotemporal dendritic voltage recordings
- SfN 2005
- abstract poster talk Model-based optimal interpolation and filtering for noisy, intermittent biophysical recordings
- Cns 2006
- Model-based smoothing of noisy, intermittent biophysical samples
- Cosyne 2007
Medicine
Peer reviewed
- doi pdf Screening patients with sensorineural hearing loss for a vestibular Schwannoma using a Bayesian classifier
- J. Clin. Otolaryngology (2007) 32(4):248-54
Abstracts / Unrefereed
- Screening of patients with sensorineural hearing loss for a vestibular Schwannoma
- R. Soc. Med. Section of Otology Abs. 2006
- Multiple endocrinopathies presenting simultaneously in the post-partum phase
- SfE National Clinical Cases 2012