Serotonergic modulation of cognitive computations

Serotonin is a neuromodulator that is implicated in awakesleep cycle, motor behaviors, reward, motivation, and mood. Recent molecular tools for cell-type-specific activity recording and manipulation with fine temporal and spatial resolutions are providing unprecedentedly detailed data about serotonergic neuromodulation. These newly gained information show substantial differences in the signaling and effect of serotonergic neuromodulation depending on the projection targets. To find the common denominator for this diversity, we conjecture that the evolution of serotonergic neuromodulation originates from signaling the time and resource available for action, learning, and development.


Introduction
Neuromodulators 1 are a subset of neurotransmitters that release diffusely and produce various effects not limited to excitation or inhibition in various time scales [1,2]. Among the four major neuromodulators (dopamine, serotonin, noradrenaline, and acetylcholine), serotonin remains the most enigmatic. While many psychiatric drugs act on serotonin signaling and metabolism, serotonin's exact role in cognitive functions has been unclear. This is a clear contrast with dopamine, which has been shown to signal reward prediction errors [3,4].
Serotonin (5-hydroxytryptamine; 5-HT) is considered one of the most ancient signaling molecules in the history of life [5]. Serotonin is synthesized from the essential amino acid tryptophan and the majority is used outside of the brain for regulating blood pressure and gut movement. There are seven major types of serotonin receptors and all except 5-HT 3 type are G-protein-coupled receptors that affect intracellular signaling pathways. The major source of serotonergic projections in the brain are the raphe nuclei located on the midline of the brain stem, although those nuclei also include GABAergic, glutamatergic, and dopaminergic neurons. Different raphe nuclei project to different targets: downward to the spinal cord and upward to the entire forebrain ( Figure 1). The wide-spread projection and the variety of receptors pose major hurdles in clarifying serotonergic functions via conventional methods like pharmacological or genetic manipulations. Recently, however, the development of molecular tools for cell-type-specific optogenetic manipulation [6] and activity recording [7] has enabled fine-grained analysis of serotonergic functions ( Figure 2). Here we review such recent advances and sketch out a new picture of what serotonin does for our cognition and behaviors.

Recent discoveries by control and measurement of serotonin signaling Optogenetics
Optogenetics are methods to activate or inhibit a particular type of cells by expressing light-sensitive molecules, such as channel rhodopsin (ChR) for activation and halorhodopsin (NpHR) or archaerhodopsin (Arch) for inhibition, in transgenic animals or by virus infection [6]. Optogenetics allow cell-type-specific manipulation of neurons in a local circuit with high temporal specificity [6] (Figure 2a, b), which is difficult to do via conventional methods like electric stimulation or pharmacological manipulation. For selective manipulation of serotonin neurons, promoters like Pet1, Tph2, and SERT are commonly used. Here we review different behavioral effects of serotonin manipulation by optogenetics.

Reward and punishment
Classically, serotonin has been implicated in punishment, negative mood, and behavioral inhibition [8,9]. Consistent with this view, optogenetic stimulation of serotonin neurons in the median raphe nucleus (MRN) enhanced anxiety-like behavior in the elevated-plus maze and the effect was expressed via the dorsal hippocampus [10,11].
In contrast, Liu et al. reported that optogenetic activation of serotonin neurons in the dorsal raphe nucleus (DRN) using the Pet1-Cre mice induced place preference [12]. With Pet1 as promoter, ChR2 was expressed not only in serotonergic neurons but also in glutamatergic neurons in DRN, and the reinforcing effect was through glutamatergic projection to the dopamine neurons in the ventral tegmental area (VTA) [13,14]. Most recently, the reinforcing effect of DRN serotonin neuron was also observed by using Tph2 and SERT promoters, which are highly selective to serotonin neurons [15,16]. However, this reinforcing effect was observed to be through DRN serotonin neurons that co-release glutamate in VTA.
Recently, Iigaya et al. found in a probabilistic foraging task that mice often employed the win-stay-lose-switch strategy but, following long inter-trial intervals (ITIs), their choices adhered to standard reinforcement learning that take into account longer past action and reward sequences. Interestingly, DRN serotonin neuron stimulation enhanced the learning rate selectively after trials following long ITIs [17].

Patience
Another dimension where serotonin has been implicated is patience for delayed reward [18][19][20]. Optogenetic activation of serotonergic neurons in the DRN in Tph2-tTA:: tetO-ChR2(C128S) transgenic mice prolonged the nosepoking time when the mice waited for future rewards but not for spontaneous nose-poking, indicating that the prolonged nose-poking time was not induced by reinforcement [21]. A more recent study also confirmed that waiting for delayed rewards was enhanced by optogenetic activation of DRN serotonin neurons in SERT-Cre mice [22]. Their results also showed serotonin neurons  Serotonergic projections from the dorsal raphe nucleus (blue) and the median raphe nucleus (red). From the Allen Brain Connectivity Atlas (http:// connectivity.brain-map.org). stimulation did not induce any reinforcing effect in both conditioned place preference and real-time place preference tests. In an intertemporal choice task, optogenetic inhibition of dorsal raphe serotonergic neurons at the decision point promoted impulsive choice, whereas optogenetic activation had the opposite effect [23]. Activation of DRN serotonergic neurons also increased active coping with inescapable stress in rats and mice in a time-locked manner [24].
Our recent experiments with the waiting task showed that the enhancement of reward waiting by DRN serotonin neuron stimulation is context-dependent; the effect is maximal when the probability of reward delivery is high but the timing of delivery is uncertain [25]. Optogenetic axon terminal stimulation further revealed different effects of serotonin in different projection target areas [26 ]. Intriguingly, stimulation of serotonergic terminals in the nucleus accumbens had no effect in our waiting task, while it promoted the choice of larger later reward in the intertemporal choice task [23]. Stimulation of serotonergic terminals in the orbitofrontal cortex (OFC) was nearly as effective as the DRN stimulation, whereas stimulation in the medial prefrontal cortex (mPFC) extended waiting only when the reward timing was uncertain. These effects were hard to capture by simple reinforcement learning considering average discounted reward. Thus we proposed a Bayesian decision scheme in which mice had an internal model of reward delivery timing and reproduced the experimental results by assuming that DRN serotonin stimulation causes overestimation of the prior probability of reward delivery, or confidence in reward acquisition [25,26 ].

Motor behavior
The above studies on patience also showed that prolonged waiting was not due to simple behavioral inhibition, as locomotion speed while the mice headed to the 118 Computational cognitive neuroscience  food site was not affected [21]. However, acute activation of DRN serotonin neurons did suppress spontaneous locomotion, while after repeated stimulations over three weeks, spontaneous locomotion was persistently increased [27]. Activation of DRN serotonin neurons increased the number of nose-poking behaviors in a probabilistic foraging task [28 ].
Recently, Yoshida et al. showed that ventral hippocampus neurons were suppressed and MRN serotonin neurons were activated when mice were engaged in reward-seeking or punishment-avoiding lever press actions [29 ]. They further showed that by optogenetic activation of MRN serotonin neurons, ventral hippocampus pyramidal neurons were suppressed via 5-HT 3A receptors expressed in inhibitory neurons.

Social behavior
In autistic model mice with a copy number variation on chromosome 16p11.2, DRN neuron activity was found to be reduced and optogenetic stimulation of DRN serotonin neurons increased social interactions through neuron projection to the nucleus accumbens [30].

Photo-tagging
Classically, putative serotonin neurons were identified by broad spikes, slow regular firing, and suppression by 5-HT 1A receptor antagonist [19]. However, it has been difficult to precisely identify serotonergic neurons using these criteria [31]. Response diversity in the DRN may reflect non-selective recording of both serotonin and nonserotonin neurons. By combining optogenetic stimulation of serotonergic neurons and micro electrode recording with a criterion of strong, short latency excitation, it is possible to dissociate serotonin neurons and non-serotonin neurons [32]. Using this photo-tagging method, various responses of serotonergic neurons have been reported, such as conditioned cues [12,32]

Fiber photometry and micro-endoscopic imaging
Genetically encoded calcium indicator (GECI), such as the GCaMP, changes its fluorescence in response to calcium influx to the cell. Fiber photometry using GECI enables monitoring the activity of genetically defined subpopulations of neurons by implanting an optic fiber in freely moving animals. The first report of fiber photometry of DRN serotonin neural activity found that rewards including sucrose, food, sex, and social interaction rapidly activate serotonin neurons, but aversive stimuli including quinine and footshock do not [33 ]. They also reported in Pavlovian conditioning that serotonin neurons gradually developed a slow ramp-up response to the reward-predicting cue, and remained responsive to the reward [34].
Matias et al. reported that 5-HT neuron activity in the DRN encoded both positive and negative reward prediction errors in reversal learning [35]. Interestingly, this negative prediction error response was not observed in Zhong's study when predicted reward was omitted. Yoshida et al. reported that MRN but not DRN serotonin neurons exhibited sustained increased activity during sustained lever pressing for rewards [29 ].
While fiber photometry enables the measurement of average neural activities in the local circuit, recent developments in micro-endoscopic imaging allows the monitoring of individual neuron activities from deep brain structures like the dorsal raphe nucleus. Seo et al. . They performed fiber photometry in the basal forebrain and found robust increase in baseline activity with bursts and dips during wakefulness, low baseline but rhythmic activity in non-REM sleep, and quiescence in REM sleep.

Fine grain anatomy
Cell-type-specific and pathway-specific gene expression methods have remarkably advanced our knowledge of the serotonergic circuit. Watabe-Uchida and colleagues used the rabies-virus-based tracing method to analyze the mono-synaptic inputs to serotonergic neurons in the dorsal raphe (DR) and median raphe (MR) nuclei and compared them with the inputs to dopaminergic neurons in the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) [46,47]. There was substantial overlap in the input distributions to DR and MR and both received strong input from the hypothalamus. While DR received stronger inputs from the basal ganglia, amygdala, and the dopaminergic nuclei SNc and VTA, MR received stronger inputs from the hippocampus, the habenula, and midbrain nuclei, such as interpeduncular nucleus and pontine reticular nucleus. Both SNc and VTA received stronger inputs from DR than from MR. These differences suggest serotonin neurons in DR and MR play different roles in response to rewarding and aversive situations.
Ren et al. analyzed the output projection from DR by combining retrograde tracing from cortical and subcortical areas and selective staining of serotonergic and glutamatergic neurons in DR [48 ]. While serotonergic neurons projecting to cortical areas like orbitofrontal cortex (OFC) and entorhinal cortex are located in the ventral portion of DR, those projecting to subcortical areas like the central amygdala (CeA) and lateral habenula are located in the dorsal portion of DR. While most cortical projecting serotonergic neurons co-express glutamates, only a minority of subcortical projecting serotonergic neurons coexpress glutamates. Pathway-specific fiber photometry revealed that both OFC and CeA projecting serotonin neurons responded positively to reward. Interestingly, while OFC projecting serotonin neurons responded negatively to aversive stimulus, CeA projecting serotonin neurons responded positively. Furthermore, chemogenetic activation of CeA projecting serotonergic neurons increased anxiety in open field and plus maze tasks.
These recent observations by pathway and cell-typespecific anatomy and physiology show that the serotonergic system is by no means uniform, with different messages and effects depending on the projection targets.

Updated theory of serotonin
Given the diversity of the inputs, outputs, activities, and effects of serotonergic neurons, how can we present any coherent computational model of serotonergic signaling? Previous theories include punishment prediction as the opponent of dopamine [8], behavioral inhibition [49 ], temporal discounting in reinforcement learning [18], and shortcut in model-based search [50]. An evolutionary viewpoint is vital in exploring the overreaching feature of serotonergic signaling [51]. Tryptophan, the precursor of serotonin, is unique among the 20 amino acids composing proteins in that it is the largest, most hydrophobic, and least used amino acid with only one codon. An important feature of tryptophan is that it absolves blue light to emit an electron, which is used for photosynthesis in chlorophyll and light detection in rhodopsin ( Figure 3). The availability and release of serotonin are also affected by light, making serotonin levels higher during the day than at night and in summer than in winter. Light therapy is a common treatment for seasonal depressive symptoms [52], through retinal-raphe pathway or increased production of serotonin by skin exposure to light [53]. Both in plants and animals, the availability of light is a major factor for the modulation of behaviors.

Current Opinion in Behavioral Sciences
Tryptophan and its derivatives for light processing and neuromodulation [51].
As a common denominator of the diverse effects of serotonin, we propose a hypothesis that serotonin signals the availability of time and resources. Table 1 summarizes the expected modulation of behavioral parameters by serotonin. For both development and metabolism, serotonin is known to have facilitatory effects. In the reinforcement learning paradigm, availability of time enables gradual learning, more exploration, and long-term prediction. Little available time may promote quick actions and risky choices. In the model-based paradigm, more time for computation favors model-based strategy, with wider and deeper search, and more time spent for collecting evidence. This view is in general agreement with a recent proposal that serotonin signals 'beneficialness' of the current state [54].

Conclusion
The study of serotonin is often likened to 'blind men touching an elephant' [69]. Recent molecular tools for cell-type-specific activity recording and manipulation with fine temporal and spatial resolutions are providing unprecedentedly detailed data about serotonergic neuromodulation. Serotonin neurons in different nuclei or even within the same nucleus carry different information [48]. Is it a bad idea then to try to attribute a unified message or function for serotonergic neuromodulation?
Here we proposed a conjecture that the evolutionary origin of serotonergic neuromodulation is signaling the time and resource available for action, learning, and development. The necessity of different responses to this signal by different subsystems of the nervous system may be the reason why so many different receptors and intracellular signaling mechanisms came into being. As projection topography evolved to cover widespread targets, different subgroups of serotonin neurons would have fine-tuned their messages to better cater to their own recipients.
The functions and references summarized in Table 1 cover just a subset of the vast studies on serotonin and there are many observations that are not consistent with this view. Further studies focusing on time and resource availability are needed to test this conjecture. Constructive approaches by simulated or robotic agents with resource constraints would also be helpful for assessing the behavioral needs for neuromodulation [70,71].

Conflict of interest statement
Nothing declared. Science 2019, 363:538-542. This is the first report of micro-endoscopic imaging of individual serotonin neuron activities in DRN. The spatial or individual differences in activity profiles are yet to be reported, such that the observations are similar to those from population recording by fiber photometry.