| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
REVIEW
1Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health, Bethesda, Maryland; and 2Laboratory for Mathematical Neuroscience, RIKEN Brain Science Institute, Saitama, Japan
Submitted 4 April 2005; accepted in final form 23 September 2005
ABSTRACT
Expectation of reward motivates our behaviors and influences our decisions. Indeed, neuronal activity in many brain areas is modulated by expected reward. However, it is still unclear where and how the reward-dependent modulation of neuronal activity occurs and how the reward-modulated signal is transformed into motor outputs. Recent studies suggest an important role of the basal ganglia. Sensorimotor/cognitive activities of neurons in the basal ganglia are strongly modulated by expected reward. Through their abundant outputs to the brain stem motor areas and the thalamocortical circuits, the basal ganglia appear capable of producing body movements based on expected reward. A good behavioral measure to test this hypothesis is saccadic eye movement because its brain stem mechanism has been extensively studied. Studies from our laboratory suggest that the basal ganglia play a key role in guiding the gaze to the location where reward is available. Neurons in the caudate nucleus and the substantia nigra pars reticulata are extremely sensitive to the positional difference in expected reward, which leads to a bias in excitability between the superior colliculi such that the saccade to the to-be-rewarded position occurs more quickly. It is suggested that the reward modulation occurs in the caudate where cortical inputs carrying spatial signals and dopaminergic inputs carrying reward-related signals are integrated. These data support a specific form of reinforcement learning theories, but also suggest further refinement of the theory.
INTRODUCTION
The initiation of body movements can be influenced by many factors, including sensory inputs, memory, attention, and arousal. Another factor that is vital for the animal's survival is reward. Unlike the other factors, which are usually based on the past events and the present context, reward is conceived as a goal that is to be achieved in the future. This is the very nature that endows the animal with the ability to acquire a large variety of voluntary behaviors. Not surprisingly, the relationship between reward and behavior has been a central theme in psychology (Balleine and Dickinson 1998
) and has now become a main theme of neuroscience. On one hand, the retroactive nature of reward (i.e., reward influences the action that leads to the reward) still challenges experimental and theoretical neuroscientists working on neuronal mechanisms of reinforcement learning (Barto 1994; Dayan and Balleine 2002; Reynolds and Wickens 2002; Schultz et al. 1997). On the other hand, the universal feature of reward-oriented behavior among animal species including humans has provoked interdisciplinary interactions between neuroscience and animal ecology (Sugrue et al. 2005) as well as human economics (Glimcher and Rustichini 2004; Montague and Berns 2002).
What is fascinating about these theoretical advances is that they have received various lines of experimental support. For example, studies using trained monkeys revealed that many neurons in what are usually called cognitive or sensorimotor areas are modulated by expected reward. They include the dorsolateral prefrontal cortex (Barraclough et al. 2004; Inoue et al. 1985; Kobayashi et al. 2002a; Leon and Shadlen 1999; Watanabe 1996; Watanabe et al. 2002), posterior parietal cortex (Glimcher 2001; Platt and Glimcher 1999; Sugrue et al. 2004), premotor cortex (Roesch and Olson 2003, 2004), and dorsal striatum (Cromwell and Schultz 2003; Hollerman et al. 1998; Kawagoe et al. 1998; Lauwereyns et al. 2002a,b; Takikawa et al. 2002a; Tremblay et al. 1998; Watanabe et al. 2003). Furthermore, other factors that influence behavior, such as learning (Dayan and Balleine 2002), memory (Baxter and Murray 2002), and attention (Maunsell 2004), seem dependent on reward. Similar findings have been reported in functional imaging studies using human subjects (for review see McClure et al. 2004b). These data appear to provide a strong background for understanding the neural mechanism of reward-modulated behavior.
Ironically, however, the widespread effects of reward in the brain make it more difficult to pinpoint the mechanism of reward-modulated behavior. Reward-dependent neural activity may not necessarily indicate that the neurons participate in the modulation of behavior based on reward. Part of the brain that may be particularly suited for controlling or generating reward-oriented behavior is the basal ganglia. The basal ganglia receive substantial reward-related information and strongly influence body movements. First, the motor function of the basal ganglia is illustrated by various kinds of movement disorders (e.g., inability to initiate or suppress movements) in basal ganglia dysfunctions (e.g., Parkinson's disease) (Denny-Brown 1962). This "motor" function is achieved by the outputs of the basal ganglia to the brain stem motor areas (e.g., superior colliculus) (Grillner et al. 2005; Takakusaki et al. 2004) and the "movement-related " areas in the cerebral cortex through the thalamus (Hoover and Strick 1999; Parent and Hazrati 1995). Second, the reward-related information to the basal ganglia is likely to be derived from substantial inputs from the limbic system (e.g., amygdala) to the ventral striatum (e.g., nucleus accumbens) (Haber and McFarland 1999; Mogenson et al. 1980), dorsal striatum (i.e., caudate nucleus and putamen) (Ragsdale and Graybiel 1988; Uno and Ozawa 1991), and to dopamine neurons in and around the substantia nigra (Fudge and Haber 2000). In particular, dopamine neurons, which appear to carry an essential signal for reward-based learning(Schultz 1998), are an important part of the basal ganglia system and project most heavily within the basal ganglia. Third, sequential and parallel inhibitory connections in the basal ganglia are thought to be suitable for the selection and learning of optimal behavior (Hikosaka et al. 1993a; Mink 1996). Fourth, the basal ganglia are thought to play an important role in learning of sensorimotor procedures or habits (Graybiel 1998; Nakahara et al. 2001; Packard and Knowlton 2002; Salmon and Butters 1995). Finally, sensorimotor-cognitive signals originating from the cerebral cortex are funneled through the basal ganglia and are returned to the cerebral cortex (Parent and Hazrati 1995). In short, the basal ganglia are located in a perfect position to control motor behaviors based on reward information.
In the present review, we first summarize findings that support that the basal ganglia are involved in reward-oriented behavior. We will then focus on studies using saccadic eye movement, which provides an excellent experimental model. Neural network models based on these studies may be applicable to other motor behaviors. Finally, the results will be discussed in a wider and theoretical framework, especially in relation to reinforcement learning theories.
ROLE OF THE BASAL GANGLIA IN REWARD-ORIENTED BEHAVIOR
The relationship of the basal ganglia with reward-oriented behavior may not be evident at a first glance. Severe movement disorders commonly found in patients with dysfunctions of the basal ganglia, such as Parkinson's disease and Huntington's disease, demonstrate that the basal ganglia play a key role in controlling body movements (DeLong and Georgopoulos 1981; Denny-Brown 1962). However, anatomical and physiological studies cast doubt on this motor-only view of the basal ganglia. The motor functions appear to be represented in limited parts in the basal ganglia, specifically, the caudal part of the putamen and related areas that receive heavy projections from the motor areas in the cerebral cortex (Flaherty and Graybiel 1995; Künzle 1975; Takada et al. 1998). A significant portion of the basal ganglia receives inputs mainly from association cortices (Parent and Hazrati 1995; Selemon and Goldman-Rakic 1985; Yeterian and Pandya 1991) and appears to be devoted to nonmotor cognitive functions (Brown et al. 1997; Middleton and Strick 1994; Packard and Knowlton 2002). The caudate nucleus, another part of the dorsal striatum, is a key station of the cognitive functions and in addition controls saccadic eye movement (Hikosaka et al. 2000). However, the relationship to reward is not evident in this scheme of the basal ganglia.
Reward has been discussed mainly in relation to the ventral striatum and/or the nucleus accumbens (Mogenson et al. 1980; Swanson 2000). It receives massive inputs from the limbic system, especially the amygdala (Fudge et al. 2002) and the orbitofrontal cortex (Haber et al. 1995; Selemon and Goldman-Rakic 1985). Lesions in the ventral striatum result in deficits in a variety of tasks in which reward-predictive cues are used to guide subsequent responding (Everitt et al. 1991, 1999; Kelley 2004). Neurons in the ventral striatum respond to cues that predict appetitive outcomes (Nicola et al. 2004; Setlow et al. 2003). These findings, together with anatomical data, gave rise to the view that motor, oculomotor, cognitive, and motivational functions are served separately by independent loop circuits formed by subareas in the basal ganglia and subareas in the cerebral cortex (Alexander et al. 1986). However, subsequent anatomical studies have shown that there are indirect pathways that originate from the ventral striatum and reach the dorsolateral striatum through the mutual and partially overlapping connections between the striatum and midbrain dopamine neurons (Haber et al. 2000).
Indeed, recent studies using monkeys and humans have revealed that the impact of reward is almost equally as strong in the dorsal striatum as in the ventral striatum. This was shown by manipulating the amount, frequency, or kinds of reward depending on task context while the subject is engaged in a sensorimotor task. Activity of single neurons in the dorsal as well as ventral striatum is strongly influenced by the expected outcome of food or water reward (Hollerman et al., 1998) (Fig. 1). For example, many neurons in the caudate and putamen exhibit sustained activity before an expected reward is delivered (Hikosaka et al. 1989c), similarly to neurons in the ventral striatum (Schultz et al. 1992). Other neurons respond differentially to sensory stimuli that indicate the presence or absence of upcoming reward (Hollerman et al. 1998; Kawagoe et al. 1998). Such anticipatory and sensory responses are related to the amount of the expected reward (Cromwell and Schultz 2003) or to the temporal proximity of reward delivery (Bowman et al. 1996). These types of reward-related activity are found in the dorsal striatum as often as in the ventral striatum (Cromwell and Schultz 2003; Hollerman et al. 1998). In the rat similar reward-predicting activity is found in the ventral striatum (nucleus accumbens) (Miyazaki et al. 1998; Nicola et al. 2004), but the dorsal striatum has not been examined in this respect.
|
; Knutson et al. 2001; Ullsperger and von Cramon 2003). Some of them also revealed activation of the dorsal striatum (Delgado et al. 2000; O'Doherty et al. 2002). In some cases significant activation was found preferentially in the dorsal striatum (Haruno et al. 2004).
Clearly, the reward-related processes are not localized in the ventral striatum but are prevalent in the entire striatum. One qualitative difference may be that many neurons in the dorsal striatum exhibit sensorimotor or cognitive activities and these activities are modulated by the nature of the expected reward (Cromwell and Schultz 2003; Kawagoe et al. 1998; Watanabe et al. 2003), whereas neurons in the ventral striatum are less selective for sensorimotor events (Schultz et al. 1992) and tend to occur just before or after the delivery of reward (Hollerman et al. 1998). Thus the dorsal striatum, rather than the ventral striatum, is a place where reward-related information is integrated into specific sensorimotor/cognitive information (O'Doherty et al. 2004).
Our main goal of this review is to understand the neuronal mechanism with which behavior is influenced by reward. In this sense the dorsal striatum seems a good choice because a specific body movement can be identified to which the particular part of the dorsal striatum is related. The effect of reward and its neural mechanism can be assessed by examining the reward-dependent changes in the movement as well as in neuronal activity. Simple hand movement can be used to study the reward-related mechanism in the putamen because the putamen is known to control skeletomotor behaviors (DeLong and Georgopoulos 1981). Many studies have shown that both reaction time and movement time are significantly shorter when a big reward is expected than when a small reward is expected (Cromwell and Schultz 2003; Hollerman et al. 1998; Minamimoto et al. 2005). On the other hand, saccadic eye movement is a good behavioral measure to study the reward-related mechanism in the caudate nucleus (Takikawa et al. 2002b) because the caudate is known to control saccadic eye movement (Hikosaka et al. 2000).
In this review we focus on the mechanism of saccadic eye movement and use it as a behavioral measure of the effects of expected reward. It has been well documented that the neural circuits involving the caudate nucleus and the substantia nigra pars reticulata (SNr) exert a powerful control over the generation of saccadic eye movement (Hikosaka et al. 2000). Another advantage of using saccadic eye movement is that its mechanism in the brain stem has been studied perhaps more extensively than any other motor behavior (Sparks 2002). Furthermore, an important component of reward-modulated behavior is orienting movement (Swanson 2000) and an important component of orienting movement is saccadic eye movement (Hayhoe and Ballard 2005). The animal must orient its eye, head, and body to the location where reward is available before procuring it (Ewert 1980). Saccadic eye movement is particularly important for humans (Johansson et al. 2001; Triesch et al. 2003) and monkeys (Miyashita et al. 1996). As will be shown herein, the initiation of saccadic eye movement is clearly facilitated or suppressed depending on the reward outcome.
BASAL GANGLIA CONTROL SACCADIC EYE MOVEMENTS WITH INHIBITORY CONNECTIONS
In this section we describe the neuronal network in the basal ganglia that controls saccadic eye movement. The basal ganglia control saccadic eye movement with the GABAergic inhibitory connection from the SNr to the superior colliculus (SC) (Fig. 2A). SNr neurons project predominantly to the intermediate layer of the SC (Chevalier et al. 1984; Graybiel 1978; Hikosaka and Wurtz 1983b; Jayaraman et al. 1977; May and Hall 1984) and have synaptic connections with saccadic burst neurons or buildup neurons (Karabelas and Moschovakis 1985). Because most SNr neurons are spontaneously very active, firing tonically and rapidly, SC saccadic neurons are kept inhibited by the output of the basal ganglia (Fig. 2B). Importantly, a majority of SC-projecting SNr neurons decrease or cease firing before a saccade directed to the contralateral side (Hikosaka and Wurtz 1983b; Joseph and Boussaoud 1985). This leads to a decrease of inhibition (disinhibition) on SC saccadic neurons (Chevalier et al. 1985) and gives a strong thrust for SC neurons to fire intensely (Fig. 2B).
|
; Wurtz et al. 2001) including the cerebral cortex, cerebellum, and brain stem. Inputs from the cerebral cortex originate mainly from the so-called cortical eye fields including the frontal eye field (FEF) (Segraves and Goldberg 1987; Sommer and Wurtz 2000), supplementary eye field (SEF) (Shook et al. 1991), and parietal eye field (PEF) in the area LIP (Paré and Wurtz 2001). Notably, all these areas contain neurons that fire before a saccade. In short, SC saccadic neurons are controlled by excitatory and inhibitory inputs from multiple sources. Before saccade, SC neurons receive excitatory inputs from the cortical eye fields while they are freed from the tonic inhibition from SNr neurons (Fig. 2A). The removal of the tonic inhibition (disinhibition) effectively opens the gate for the saccadic output.
SNr neurons cease firing before a saccade because they are inhibited by neurons in the caudate nucleus (CD) (Fig. 2B). Output neurons in the basal ganglia, which are located in the SNr and the internal segment of the globus pallidus (GPi), are innervated directly by projection neurons in the striatum (caudate and putamen) (François et al. 1994). This direct innervation is also GABAergic and inhibitory (Fonnum et al. 1978; Kita 1993; Yoshida and Precht 1971). Projection neurons in the caudate are characterized anatomically as medium-spiny neurons (Preston et al. 1980). They have deep membrane potentials at rest and exhibit action potentials only occasionally (Wilson and Kawaguchi 1996). In the caudate of awake animals, neurons with very low spontaneous activity are considered to be projection neurons (Hikosaka et al. 1989a; Kimura et al. 1990). Many of them exhibit a burst or a train of spikes in relation to saccadic eye movements, but otherwise are nearly silent (Hikosaka et al. 1989a). Electrical stimulation of the saccade-related area in the caudate induces an inhibition of saccade-related SNr neurons (Hikosaka et al. 1993b). These results suggest that the saccade-related decrease in firing in SNr neurons is caused by the saccade-related increase in firing of caudate neurons.
The main inputs to the caudate originate from various areas in the association cortex (Selemon and Goldman-Rakic 1985; Yeterian and Pandya 1991) and the thalamic nuclei related to the oculomotor areas (Takada et al. 1985). Saccade-related neurons are found in the part of the caudate that receives inputs mainly from the FEF (Stanton et al. 1988), SEF (Parthasarathy et al. 1992; Shook et al. 1991), and the dorsolateral prefrontal cortex (Selemon and Goldman-Rakic 1985; Yeterian and Pandya 1991). Therefore saccade-related activity of caudate neurons is probably caused by excitatory inputs from these cortical areas.1
BASAL GANGLIA GUIDE SACCADE TO WHERE REWARD IS AVAILABLE
A feature common to caudate and SNr neurons is that their activity is often strikingly dependent on the behavioral context. For example, some caudate neurons respond to a visual stimulus only when its position must be remembered (Hikosaka et al. 1989b); other caudate neurons fire before a saccade only when it is guided by memory (Hikosaka et al. 1989a). Such strong context dependency is also found in SNr neurons (Hikosaka and Wurtz 1983a). Another feature is that many caudate neurons fire tonically in an anticipatory manner before an expected task-related event occurs, such as before the onset of an expected target or the delivery of an expected reward (Hikosaka et al. 1989c; Hollerman et al. 1998; Nishino et al. 1984; Rolls et al. 1983). "Reward" turned out to be a key factor that characterizes the information processing in the basal ganglia, as described below.
To examine the effect of reward on saccadic eye movement, we used saccade tasks in which the amount of reward is unequal among possible target positions. We chose position as a cue for reward for two reasons. First, the goal of saccadic eye movement is to localize an object in space (Hayhoe and Ballard 2005; Johansson et al. 2001). Second, when an animal forages for food, the most crucial behavior is to localize a place where the food is available (Swanson 2000). Positional cues have widely been used in learning tasks such as conditioned place preference task (Carr and White 1983; Everitt et al. 1991; Spyraki et al. 1982).
In our saccade tasks the target was presented randomly at one out of two or four directions, but only one direction was associated with a big reward, whereas the others were associated with a small or no reward (Fig. 3A). The big-reward direction was fixed in a block of 20–60 trials and is changed in the next block (Fig. 3B). Let us call this 1DR (one-direction rewarded) saccade task. We used visual and memory versions. In the visual-1DR task, the monkey makes a saccade to the target immediately after its onset (Fig. 3A, top) (Lauwereyns et al. 2002b). In the memory-1DR task, the target position was cued and the monkey has to make saccade to the cued position after a time delay based on memory (Fig. 3A, bottom) (Kawagoe et al. 1998).
|
; Watanabe and Hikosaka 2005). In the schematic example shown in Fig. 3A, top, the latencies of rightward saccades are shorter in the reward condition R (rightward saccades associated with bigger rewards than leftward saccades) than in the condition L (leftward saccades associated with bigger rewards). Conversely, the latencies of leftward saccades are shorter in condition L than in condition R. During the experiment the reward conditions R and L were alternated usually every 20 trials with no external instruction (Fig. 3B), and the saccade latency changed reliably (Fig. 3C). One interesting finding common to all monkeys tested was that saccade latency decreased quickly during a small-to-big reward transition and increased more slowly during a big-to-small reward transition (Fig. 3C) (Watanabe and Hikosaka 2005). Similarly, in the memory-1DR task, latencies were shorter and peak velocities were higher when the saccades were big-rewarded than when small-rewarded (Takikawa et al. 2002b). The variations of saccadic velocity, latency, and amplitude were smaller in the big-reward condition (Takikawa et al. 2002b). Thus saccadic eye movement provides a solid and reliable behavioral measure to study the neural mechanisms of reward-oriented behavior. A series of experiments, reviewed below, suggest that the basal ganglia play a key role in the reward-dependent modulation of saccades. 2
In the visual-1DR task many caudate projection neurons exhibit tonic activity (Fig. 4) (Lauwereyns et al. 2002b). In each trial a central spot came on and the monkey had to fixate it. The caudate neurons then started firing (Fig. 4A). The firing rate increased gradually until the fixation point went off (and a target appeared). A unique feature of the activity was that it occurred selectively or preferentially when one particular position (usually contralateral to the recording site) was rewarded (Takikawa et al. 2002a). Among caudate neurons that exhibited precue tonic activity, a majority showed the selectivity to the reward position. This kind of position selectivity is different from visual receptive field because the neuronal activity occurs before any target comes on. It is different from saccade movement field (Sparks et al. 1976; Wurtz and Goldberg 1972) or memory field (Funahashi et al. 1989; Williams and Goldman-Rakic 1995) because saccade direction is not specified during the fixation period. This reward-position–selective activity had not been reported in other brain areas, but subsequently was found in the SNr (Sato and Hikosaka 2002) and SC (Ikeda and Hikosaka 2003), which receive inputs from the caudate.
|
), the SNr–SC connection is predominantly ipsilateral (Beckstead et al. 1981), and the SC–brain stem connection is predominantly contralateral (Grantyn et al. 1979). Therefore a change in caudate neuronal activity affects contralateral saccades predominantly (Itoh et al. 2003). Figure 5 shows the caudate–SNr–SC pathways in both hemispheres.
|
). The decrease in activity of SNr neurons then leads to a disinhibition of left SC neurons (Ikeda and Hikosaka 2003). The SC neurons should then be more excitable, although they may not fire. Second, consider a right caudate neuron with typical reward-position–selective activity. Because the big-reward side (i.e., right) is ipsilateral to the neuron, the caudate neuron should be less active. SNr neurons on the right side are then not inhibited and consequently right SC neurons are kept inhibited by SNr neurons. What we see here is a strong bias in excitability between SC neurons on the left and right sides. Note that this bias occurs before any motor instruction (i.e., target) comes in (Fig. 5, top). If a target appears on the right side (white dot in Fig. 5, bottom, right), the visual signal is registered mainly in the left cortical areas (yellow circle), which then activates the left SC. Because the excitability of the left SC neurons has been elevated by the reward-position–selective activity derived from the left caudate, the SC neurons respond quickly to the cortical input and consequently a saccade occurs quickly to the right target. This is where a big reward is available. In contrast, if a target appears on the left side (Fig. 5, bottom, left), the visual signal is registered in the right cortical areas, which then activates the right SC. Because the excitability of the right SC neurons has remained depressed, they respond slowly to the cortical input and consequently a saccade occurs slowly to the left target where a small or no reward is available. This is exactly what happened in the visual-1DR task (Fig. 4B).
When the big-reward side is changed to the left in the next block of 1DR, the situation is completely reversed. Right caudate neurons exhibit higher reward-position–selective activity and therefore left saccades occur quickly, whereas left caudate neurons exhibit weaker reward-position–selective activity and therefore right saccades occur slowly.
These data suggest that the reward-position–selective activity in caudate neurons contributes to the emergence of the reward-dependent bias of saccade latency, that is, quicker saccade to a big reward and slower saccade to a small reward. This hypothesis is supported by the findings indicating that the firing rates of reward-dependent caudate neurons are correlated with saccade velocity and latency (Itoh et al. 2003; Watanabe et al. 2003). The causal role of caudate neurons is supported by our recent experiment showing that the reward-dependent bias of saccade latency is attenuated by the blockade of dopamine-D1 receptors in the caudate (Nakamura and Hikosaka 2004).
NEURONAL ACTIVITY IN THE BASAL GANGLIA IS MODULATED BY EXPECTED REWARD
In the preceding sections we have indicated that caudate neurons create inclination of saccades to the rewarded position before any instruction is given. In many situations, there is also a preparatory process before an action (saccade) during which some instructions or cues are presented to indicate what kind of movement is required or how rewarding the movement will be. To examine the effect of reward expectation on the preparatory process, we modified the memory-guided saccade task (Hikosaka and Wurtz 1983a) such that only one particular direction is associated with a big reward. We call it the memory-1DR task (Fig. 3A, bottom). Because the action (saccade to the cue position) as well as the reward outcome becomes predictable at the presentation of the target cue, it is very important to study the neuronal responses to the cue. If neurons are related to the preparation of reward-oriented action, their responses to the target cue should change depending on the expected reward outcome. Indeed, many caudate and SNr neurons exhibit such reward-predictive responses.
Kawagoe et al. (1998) indeed found that visual responses of caudate projection neurons to the cue were strongly modulated by the expected reward, as exemplified in Fig. 6A. The neuron responded most strongly to the cue on the left (L), which was contralateral to the neuron, when the reward was given equally for different target positions (Fig. 6A, rightmost column, indicated by ALL). However, the neuron's responses to the cues changed dramatically in four blocks of the memory-1DR task (Fig. 6A, left four columns), in each of which reward was given for only one direction out of four. Compared with the case where an equal amount of reward was given for all four targets (ALL), the neuron's response in the 1DR task was enhanced when the cue indicated an upcoming reward (e.g., cued direction: U; rewarded direction: U), whereas it was depressed when the cue indicated no upcoming reward (e.g., cued direction: U; rewarded direction: R, L, or D). The reward modulation was so strong that the neuron's original direction selectivity was shifted or even reversed. Overall, statistically significant modulation by the expected reward was observed in about 80% of visually responsive caudate neurons.
|
). About half of the task-related SNr neurons exhibited decreases in activity, whereas the other half exhibited increases in activity. The decrease in activity is likely to be caused by the direct GABAergic action of caudate neurons (Fig. 2). The increase in activity may arise from the indirect effect mediated by GPe neurons and possibly by subthalamic nucleus (STN) neurons as well (not shown). The decreasing type neurons were usually selective in space (responding to contralateral cues) and in reward (activity decreased strongly in the rewarded trials) than the increasing type neurons. These results are in agreement with the hypothesis that the basal ganglia contribute to the selection of action by removing the tonic inhibition selectively on the action to be selected (using the direct pathway) while increasing the inhibition on a wide variety of actions (using the indirect pathway) (Mink 1996).
How have the caudate and SNr neurons come to acquire activity dependent on expected reward? One possibility is that caudate neurons receive signals from other brain areas that have already been modulated by expected reward, as described in the INTRODUCTION. A second possibility is that the reward modulation first occurs in the basal ganglia. In this case, the cerebral cortex would receive reward-modulated signals from the basal ganglia through the thalamus (Middleton and Strick 2000). A recent study by Pasupathy and Miller (2005) seems consistent with this idea. In the following we will examine the hypothesis that the reward modulation occurs, at least partly, in the basal ganglia depending on the inputs from dopamine neurons.
The caudate is heavily innervated by axons releasing "modulators" such as dopamine (DA) and acetylcholine (ACh), which might carry reward-related signals (Chiara et al. 1994; Graybiel 1990). That dopamine is essential for normal functions of the basal ganglia is demonstrated by the fact that Parkinson's disease is caused by the loss of dopamine innervation in the striatum (putamen and caudate) (Selby 1968). A pure dopamine deficit can be induced experimentally by MPTP (1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine) and this model has been used widely. Human patients with MPTP-induced parkinsonism showed severe deficits in saccadic eye movements (Hotson et al. 1986). After a local and unilateral injection of MPTP into the caudate, monkeys exhibited contralateral hemineglect (Miyashita et al. 1995) and eye movements during visual search became restricted in the hemifield ipsilateral to the MPTP injection (Kato et al. 1995). Moreover, saccades became infrequent, small in amplitude, and slow; the visual search area became smaller; and the duration of eye fixation became longer. These results indicate that dopamine in the caudate is capable of modulating saccadic eye movements, but do not indicate in what context it does so.
The relation of dopamine to reward was originally proposed based on the "self-stimulation" experiments (Olds and Milner 1954). If an electrode is placed in a particular part of the brain and the animal itself is allowed to stimulate the part of the brain by pressing a key, the animal may press the key continually and obsessively. This effect is stronger if the electrode is placed closer to the ascending axon bundle of midbrain dopamine neurons (Corbett and Wise 1980). The effect is blocked if dopamine transmission is blocked in the striatum by injecting a dopamine antagonist (Mora et al. 1976; Phillips et al. 1976). A second line of evidence comes from the studies on drug addiction (Berke and Hyman 2000; Wise 1996). A variety of drugs, which induce addiction, effectively increase the extracellular concentration of dopamine in the striatum (Berke and Hyman 2000), especially the ventral striatum or nucleus accumbens.
More direct evidence for the relationship of dopamine to reward is based on single-unit studies on midbrain dopamine neurons in trained animals. Schultz and colleagues demonstrated that dopamine neurons respond to the delivery of reward (Schultz et al. 1993). A key finding was that the response was correlated with the difference between the expected reward and the actual reward. Thus the response to a reward is stronger if it is not expected (Mirenowicz and Schultz 1994). The response is negative (i.e., a decrease in firing) if the expected reward is not given (Hollerman and Schultz 1998). In short, dopamine neurons appear to encode "reward prediction error" (Schultz 1998). This signal corresponds nicely to a principal factor in learning theories that account for classical conditioning (Rescorla and Wagner 1972) as well as in reinforcement learning theory, which was developed more recently (Barto 1995; Dayan and Balleine 2002; Houk et al. 1995; Schultz and Dickinson 2000) (see RELATION TO REINFORCEMENT LEARNING THEORIES).
Dopamine neuronal activity in the memory-1DR task followed this principle (Kawagoe et al. 2004). In an animal trained extensively in 1DR, dopamine neurons responded to the cue with a phasic increase in firing if the cue indicated an upcoming reward; they responded to the cue with a phasic decrease in firing if the cue indicated no reward (Fig. 6B). Dopamine neurons showed no spatial selectivity: they responded to the cue at any position equally as long as it indicates an upcoming reward or as long as it indicates no reward. Dopamine neurons exhibited no response to the cue when all positions were equally rewarded (Fig. 6B, ALL). Dopamine neurons thus represented the difference between the expected value of reward cue and the actual value of reward cue, consistent with an extension of the "reward prediction error" theory (Tobler et al. 2003). Before the cue is presented, the likelihood of obtaining reward in the current trial is 25% (1 out of 4), but it increases to 100% after the reward-indicating cue, but decreases to 0% after a nonreward-indicating cue. Dopamine neurons' "positive" and "negative" responses are correlated respectively with the increase and decrease in reward prediction. If all positions are equally rewarded, the likelihood of reward is 100% before the cue and this is not changed by the presentation of the cue; thus dopamine neurons show no response.
Here we see an intriguing relationship between caudate projection neurons and dopamine neurons (Fig. 6) (Kawagoe et al. 2004). In both caudate and dopamine neurons, the visual responses in 1DR task are strongly modulated by the expected reward. They are different in that caudate projection neurons, not dopamine neurons, show spatial selectivity. These results suggest mutual interactions between caudate projection neurons and dopamine neurons. In the next section we first consider the influence of dopamine activity on caudate activity and later the influence of caudate activity on dopamine activity.
POSSIBLE MECHANISM OF REWARD-BASED LEARNING IN THE CAUDATE
As mentioned earlier, the saccade-related neurons in the caudate receive inputs from the cortical eye fields and dorsolateral prefrontal cortex where neurons typically exhibit spatially selective visual or saccadic activities. Interactions between spatial signals from the cerebral cortex and reward-related signals from dopamine neurons may induce reward-modulated spatial signals in caudate neurons.
Physiological studies have suggested that dopamine inputs can modify cortically induced excitatory postsynaptic potentials (EPSPs) (Reynolds and Wickens 2002). It could occur instantaneously by changing the properties of many voltage-dependent ion channels (Nicola et al. 2000) or by modulating synaptic plasticity through the mechanisms of long-term potentiation (LTP) or long-term depression (LTD) (Calabresi et al. 1996; Lovinger et al. 2003; Mahon et al. 2004; Reynolds and Wickens 2002). Earlier in vitro studies on striatal projection neurons showed that LTD is common, whereas LTP is induced only when Mg is absent or dopamine is applied (Calabresi et al. 1992). More recent in vivo studies indicated that LTP is commonly induced if the recorded striatal projection neuron is depolarized (Charpier and Deniau 1997) and D1 dopamine receptors are activated (Reynolds et al. 2001). LTD is also induced in vivo if dopamine receptors are not activated (Reynolds and Wickens 2000). These findings can be interpreted by the following rules as previously proposed (Schultz 1998; Wickens and Kötter 1995), although there may be other interpretations.
RULE 1. The synaptic efficacy of the cortical input to a caudate neuron increases with LTP every time the cortical input elevates the caudate neural activity in conjunction with an increase in the dopamine input.
RULE 2. The synaptic efficacy of the cortical input to a caudate neuron decreases with LTD every time the cortical input elevates the caudate neural activity in conjunction with a decrease in the dopamine input.
Let us consider the activity of a model caudate neuron that receives two kinds of inputs from the cerebral cortex carrying visual signals from the left (L) and right (R) cues, respectively (Fig. 7). The cortical inputs make synapses probably on different sets of dendritic spines (X and Y) on the caudate neuron because the innervation of single corticostriatal axons is extremely sparse (Zheng and Wilson 2002). On the other hand, dopamine neurons are likely to have synaptic contacts diffusely with both spine X and spine Y, as suggested by anatomy (Hedreen and DeLong 1991). Suppose that reward is associated with L cue, but not R cue (Fig. 7). If the target cue appears on the left in one trial, the visual signal is fed into the caudate neuron through spine X (Fig. 7, top, left, indicated by a thick red line), which induces firing in the caudate neuron. At the same time dopamine neurons in the SNc fire because the cue indicates an upcoming reward (indicated by a thick green line). Thus Rule 1 holds for spine X and thus the efficacy of the L–X synapse should be increased. If the target cue appears on the right, the visual signal is fed into the caudate neuron through spine Y (Fig. 7, top, right), but the dopamine input is reduced because the cue indicates no reward (indicated by a thin gray line). Thus Rule 2 holds for spine Y and thus the efficacy of R–Y synapse should be decreased.
|
). In short, the model caudate neuron changes the responses to the visual cues such that it responds to whichever cue indicates an upcoming reward (Fig. 7) as the actual caudate neuron did (Fig. 6A) (Nakahara et al. 2002).
Note that, unlike the neuron shown in Fig. 6A, there are caudate neurons that maintained their direction preference (usually contralateral) even when the reward direction is changed, although their selectivity is enhanced or depressed depending on the expected reward (Kawagoe et al. 1998). Suppose such a neuron is in the left caudate; it may then have input R exclusively or preferentially. The neuron's response to input R changes depending on the reward condition, but it may not respond to input L even if it indicates an upcoming reward. The contralateral preference of caudate neurons is thus maintained as a result of the asymmetry of cortical inputs.
What is not explained by the model is the presence of caudate neurons that respond to a no-reward–indicating cue more strongly than to a reward-indicating cue (Kawagoe et al. 1998). One solution would be to change the rules (Rules 1 and 2) such that the directions of plasticity are reversed; LTP and LTD may be deployed in the opposite conditions for these negatively modulated caudate neurons (Nakahara et al. 2002). Different dopamine receptor types might mediate opposite effects in different groups of caudate neurons. In fact, D1 and D2 receptors are expressed preferentially in direct-pathway neurons and indirect-pathway neurons, respectively (Gerfen et al. 1990; Surmeier et al. 1996). It is possible that the positively and negatively modulated caudate neurons correspond to the two types caudate neurons belonging to the direct and indirect pathways. These ideas remain to be tested experimentally.
The proposed model is supported by recent experiments conducted by in our laboratory. First, we found that electrical stimulation of the caudate can induce changes in saccade latency similar to those induced by reward modulation (Nakamura and Hikosaka 2003). While the monkey was engaged in a visually guided saccade task, the saccade-related area of the caudate was electrically stimulated only after a saccade and only when the saccade was directed to a particular position. After repeating the trials, the latency became gradually shorter for the saccade that was associated with electrical stimulation and the effect remained even after the stimulation was stopped. Second, the reward modulation of saccade behavior can be changed by blockade of dopamine receptors (Nakamura and Hikosaka 2004). A D1 antagonist or a D2 antagonist was injected into the region of the caudate where saccade-related neurons are clustered while the monkey performed the visual-1DR task. D1 antagonist attenuated but D2 antagonist enhanced the reward modulation of saccade behavior. These results suggest that 1) the reward modulation of saccadic eye movement, at least partly, originates from the caudate; 2) the dopamine input to caudate projection neurons is responsible for the reward modulation; and 3) the dopamine effect is mediated by D1 and D2 receptors in differential manners.
Thus far we have discussed how dopamine inputs might modify the cortical inputs to caudate projection neurons. However, the completely opposite relationship might also be present: Activity of dopamine neurons may be modulated by signals from caudate neurons. A majority of caudate projection neurons project to the substantia nigra (Lynd-Balta and Haber 1994b; Parent and Hazrati 1994). Although their axons make synapses mainly on GABAergic neurons in the pars reticulata (SNr), some make synapses on dopamine neurons in the pars compacta (SNc) and the overlying tegmental area (Hedreen and DeLong 1991). Furthermore, many SNr neurons emit axon collaterals that then make synapses on nearby dopamine neurons (Tepper et al. 1995; Van den Pol et al. 1985). It is then likely that the reward-modulated activity of caudate neurons modulates activity of dopamine neurons. Because both caudate projection neurons and SNr neurons are GABAergic, the direct caudate–dopamine effect is inhibitory, whereas the indirect caudate–SNr–dopamine effect is disinhibitory. In addition, signals from the caudate to dopamine neurons may be transmitted through the indirect pathways involving the GPe and STN (Iribe et al. 1999; Smith et al. 1998). The responses of dopamine neurons to reward-predicting and no-reward–predicting stimuli, which are a phasic increase and a phasic decrease in firing, might be explained by these caudate-originated signals.
Note, however, that dopamine neurons in less experienced monkeys continue to respond to rewards, but not to their predictors, in the memory-1DR task. The response to the predictor (cue) starts to appear after the monkey experiences many sessions of the memory-1DR task, but initially very slowly within a given block (Takikawa et al. 2004). These results suggest that the dopamine neuronal response to predictor is acquired with long-term learning. We speculate that the caudate contributes to the learning in dopamine neurons or send learned signals to dopamine neurons through the direct and indirect connections described above.
It should be noted, however, that the mutual relationship between caudate and dopamine neurons alone may not fully account for the emergence of the reward-predicting information. In fact, both caudate and dopamine neurons receive inputs from many brain areas outside the basal ganglia. For example, caudate neurons receive inputs from the thalamus, such as the parafascicular nucleus (Sadikot et al. 1992), mediodorsal nucleus (Parent et al. 1983), and nucleus ventralis anterior (Nakano et al. 1990), in addition to the cerebral cortical areas. Dopamine neurons receive inputs from the lateral hypothalamus (Fadel and Deutch 2002), amygdala (Fudge and Haber 2000; Lee et al. 2005), nucleus accumbens (Lynd-Balta and Haber 1994b), pedunculopontine nucleus (Kobayashi et al. 2002b; Lavoie and Parent 1994; Pan and Hyland 2005), superior colliculus (Dommett et al. 2005), bed nucleus of stria terminalis (Fudge and Haber 2001; Georges and Aston-Jones 2001), and habenula (Christoph et al. 1986). The reward-predicting information might be transmitted from these areas to the basal ganglia. Alternatively, it might be generated in the basal ganglia based on different kinds of information derived from these areas.
RELATION TO REINFORCEMENT LEARNING THEORIES
A prominent problem in reinforcement learning is that a reward may occur after some time delay, not immediately after a single action but after a sequence of several actions. To maximize reward acquisition in such a case, a mechanism must be devised to learn to account for such delayed rewards and choose a sequence of actions accordingly. A prominent model of reinforcement learning uses, as a learning signal, reward prediction error, which is the difference between the expected reward (with delayed rewards taken into account) and the actual reward (Sutton and Barto 1998). This model is called "temporal difference model" or "TD model," whereas the reward prediction error is called TD error. The TD model is particularly attractive because dopamine neuronal responses are found to be similar to the simulated TD errors (Hollerman and Schultz 1998; Mirenowicz and Schultz 1994; Waelti et al. 2001).
The TD hypothesis can also explain dopamine responses in the memory-1DR task (Fig. 6B). In this task, there were four possible target positions in each trial, one rewarded and the other three nonrewarded. Then, the reward probability was 1/4 before the target cue appeared. Once the cue came on, the reward probability became either 1 (after reward-indicating cue) or 0 (after no-reward-indicating cue). Reward prediction error was then either 3/4 or –1/4. Indeed, this was the time when dopamine neurons burst or stop firing. These results are consistent with the hypothesis that dopamine neurons represent TD error (Houk et al. 1995; Montague et al. 1996; Nakahara et al. 2002; Schultz et al. 1997; Suri and Schultz 1998).
Because TD error is computed roughly as the difference between the expected reward and the actual reward, different brain areas may provide dopamine neurons with the expected reward signal and the actual reward signal. As shown previously (Fig. 6A), activity of many caudate neurons is modulated strongly by expected reward and therefore may provide dopamine neurons with the expected reward signal. Although this hypothesis has been put forward by many researchers, the detailed comparison between caudate and dopamine neuronal responses in the memory-1DR task (Fig. 6) provides a stronger support for the hypothesis.
The TD model typically has two components, called actor and critic (Barto 1995). The critic mechanism learns to predict future reward outcome, whereas the actor mechanism learns to choose an action in each trial. TD error is computed after each rewarded or nonrewarded trial and both critic and actor mechanisms adjust, albeit gradually, their states by using the TD error. It has been hypothesized that the critic mechanism is composed of the mutual connections of caudate neurons and dopamine neurons, whereas the actor mechanism is composed of other caudate neurons that also receive dopamine inputs but project to the SNr or GPi and then influence motor outputs (Barto 1995; Houk 1995; Montague et al. 1996; Schultz et al. 1997). The "actor " role of the caudate has been suggested by human neuroimaging studies (O'Doherty et al. 2004; Tricomi et al. 2004). Support for this hypothesis are the findings in studies using saccade tasks: 1) activity of reward-dependent caudate neurons is correlated with parameters of saccadic eye movement in individual trials (Itoh et al. 2003; Watanabe et al. 2003) and 2) the reward-position–dependent activity of caudate neurons is likely to cause a bias of saccadic eye movement toward a reward position (as shown in Figs. 4 and 5).
Thus our experimental evidence and argument are largely consistent with the primary role of the basal ganglia in the TD model. Yet, we also see some departures as well. First, note that in the TD model the critic weighs most on the immediate reward and progressively less on rewards in the more distant future; a reward in a distant future is discounted (Schultz et al. 1997). This mechanism can adapt to many situations in which rewards come after various time delays. This is particularly true if the probabilistic distribution of the reward delays is unchanged across trials. In the real world, however, the availability of reward may change over time in a certain pattern, either deterministically or probabilistically. In this case, the same sensory input may indicate different reward probabilities depending on its position in the pattern. The TD model, in its original proposal, did not address this issue and is not ideal for adapting to such a change of reward availability.
Interestingly, we found that dopamine neurons are capable of adapting to such a probabilistic context (Nakahara et al. 2004) (Fig. 8). In most of the experiments of 1DR tasks, the target position was pseudorandomized such that the conditional probability of reward in the current trial is lower if reward has been given in a more recent trial (Fig. 8, A and B). It was found that dopamine neurons can discriminate the differences in the conditional probability of reward. Dopamine neurons' excitatory response to a reward-indicating cue is stronger if reward has been given in a more recent trial (Fig. 8C, black), which occurs because reward is less likely and therefore more "surprising" if it occurs soon after a rewarded trial. Conversely, dopamine neurons' inhibitory response to a no-reward–indicating cue is weaker if reward has been given in a more recent trial (Fig. 8, gray), a phenomenon that can be explained by dopamine neurons' coding of reward prediction error using the conditional probability of reward (Fig. 8, D and E). This indicates that dopamine neurons can gain access to rule or context beyond sensory input and can produce reward prediction error signals differentially even to the same sensory input. A different type of context dependency has recently been reported, that is, dopamine neurons can encode relative reward values even when the mean value is changed (Tobler et al. 2005).
|
). Further, when the same monkey is engaged in a different task with a purely random reward schedule, dopamine neurons behave as the critic in the conventional TD model would behave (Fig. 8E). These results indicate that dopamine neurons, after long-term learning, can adapt themselves to various contexts and exploit them. Dopamine neurons acquire such adaptability, possibly because they receive context-dependent information from caudate neurons, as suggested in the preceding section.
Other findings may also suggest a departure from the conventional TD model. As described earlier, some caudate neurons show "negative reward modulation," such that their activity is higher when reward is not expected (Kawagoe et al. 1998). Because saccade latencies are longer and velocities are lower in such nonrewarded trials (Takikawa et al. 2002b), one interpretation is that caudate neurons with negative reward modulation suppress saccades (whereas caudate neurons with positive reward modulation facilitate saccades). However, Watanabe et al. (2003) found that saccadic caudate neurons with negative reward modulation also appear to facilitate, not suppress, saccades. This finding suggests that caudate neurons with negative reward modulation might be specialized for saccades that predict a reward at some trials later. Considering the series of trials in 1DR tasks, unrewarded trials can be viewed as steps before reaching a reward. It is then possible that the actor (and consequently the critic as well) in the basal ganglia may be composed of separate elements, each of which contributes to an action related to the reward outcome in particular steps away in future. Similar neuronal activity has been found in many brain areas by Richmond and colleagues in experiments using a visual stimulus to cue the monkeys as to their progress toward earning a reward (Bowman et al. 1996). Some neurons in the ventral striatum (Shidara et al. 1998), anterior cingulate cortex (Shidara et al. 1998), and temporal cortices (Liu and Richmond 2000) become active selectively depending on the proximity to the reward delivery. This may suggest that reward-predicting mechanisms are widely distributed in the brain (O'Doherty et al. 2004) and are differentially involved in reward prediction at different reward delays (McClure et al. 2004a; Tanaka et al. 2004). These signals may be fed into dopamine neurons and enable them to adapt to a probabilistic context (Nakahara et al. 2004).
CONCLUSIONS
A series of experiments using saccade tasks with positional bias of reward suggested that the basal ganglia exert modulatory influences on the superior colliculus so as to orient the eyes to a position where a big reward is available. The modulation starts even before a target is presented. This is done by tonic activation of caudate neurons that is selective for the position where a big reward is available (Fig. 4). This pretarget signal is transmitted to the superior colliculus through the SNr, leading to a strong imbalance in excitability between the superior colliculi until a target appears (Fig. 5). Consequently, the saccade occurs quickly to the big-reward position and slowly to the small-reward position (Fig. 4). This pretarget bias is based on internal drive or desire rather than rational judgment because it occurs even when the target is expected to occur at big- and small-reward positions with equal probability. It represents motivational processes, rather than cognitive processes. A lack of this mechanism would lead to a selective impairment of internally driven actions, which actually is commonly found in human patients having various kinds of basal ganglia dysfunction (Brown and Pluck 2000; Denny-Brown 1968; Sachs 1983).
The basal ganglia continue to exert reward-modulatory effects on motor preparation processes. Many caudate neurons fire phasically or tonically in response to the cue-indicating target position, presumably by receiving inputs from cortical areas. However, the response is enhanced or depressed depending on the expected amount of reward, leading to facilitation of saccades to the big-reward position and inhibition of saccades to the small-reward position. The reward modulation of caudate neuronal responses is likely to be caused by the concurrent inputs from dopamine neurons in the SNc (Figs. 6, 7). Similar mechanisms have been proposed on the basis of cellular level studies, but there has been little evidence that such mechanisms are at work in voluntary behavior. The results obtained using saccade tasks now provide evidence to explain reward-modulated behavior by neurophysiological mechanisms in and around the basal ganglia. They further suggest that there exists a neuronal network in the basal ganglia that operates similarly to what the reinforcement learning theories, specifically the TD model, predicts (Montague et al. 2004). On the other hand, the basal ganglia network shows greater adaptability to behavioral contexts than the current TD model can do (Fig. 8).
Finally, similar mechanisms may exist in the basal ganglia for the reward-dependent modulation of other motor behaviors because the basic architecture of neural network is common among the caudate, putamen, and ventral striatum. These nuclei share the same types of projection neurons and interneurons (Kita 1993), receive heavy inputs from midbrain dopaminergic neurons (Lynd-Balta and Haber 1994a), and their outputs are mediated mainly be GABAergic inhibitory neurons (Chevalier and Deniau 1990). Thus the principle of the neural network that serves saccadic eye movement may be valid for other kinds of motor behaviors.
ACKNOWLEDGMENTS
We thank R. Wurtz and B. Richmond for valuable comments. We also thank Y. Takikawa, R. Kawagoe, J. Lauwereyns, K. Watanabe, Y. Shimo, M. Sato, T. Ikeda, S. Kobayashi, H. Itoh, M. Sakagami, B. Coe, and M. Koizumi for contributing to this study series.
FOOTNOTES
The costs of publication of this article were defrayed in part by the payment of page charges. The article must therefore be hereby marked "advertisement" in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.
1 Skeletal movements are controlled mainly by the putamen (counterpart of the caudate), the globus pallidus internal segment (GPi) (counterpart of the SNr), and its projection to the thalamus (Hoover and Strick 1999) or brain stem motor areas (Takakusaki et al. 2004). The putamen receives inputs mainly from the motor and premotor cortical areas (Nambu et al. 2002). 
2 Skeletal movements (e.g., manual key press) also show large differences in reaction times when big and small (or no) rewards are cued (Hollerman et al. 1998; Minamimoto et al. 2005).
Address for reprint requests and other correspondence: O. Hikosaka, Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health, Bethesda, MD 20892 (E-mail: oh{at}lsr.nei.nih.gov)
REFERENCES
Alexander GE, DeLong MR, and Strick PL. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Ann Rev Neurosci 9: 357–381, 1986.[CrossRef][Web of Science][Medline]
Balleine BW and Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37: 407–419, 1998.[CrossRef][Web of Science][Medline]
Barraclough DJ, Conroy ML, and Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7: 404–410, 2004.[CrossRef][Web of Science][Medline]
Barto AG. Reinforcement learning control. Curr Opin Neurobiol 4: 888–893, 1994.[CrossRef][Medline]
Barto AG. Adaptive critics and the basal ganglia. In: Models of Information Processing in the Basal Ganglia, edited by Houk JC, Davis JL, and Beiser DG. Cambridge, MA: MIT Press, 1995, p. 215–232.
Baxter MG and Murray EA. The amygdala and reward. Nat Rev Neurosci 3: 563–573, 2002.[CrossRef][Web of Science][Medline]
Beckstead RM, Edwards SB, and Frankfurter A. A comparison of the intranigral distribution of nigrotectal neurons labeled with horseradish peroxidase in the monkey, cat, and rat. J Neurosci 1: 121–125, 1981.[Abstract]
Berke JD and Hyman SE. Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25: 515–532, 2000.[CrossRef][Web of Science][Medline]
Bowman EM, Aigner TG, and Richmond BJ. Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. J Neurophysiol 75: 1061–1073, 1996.
Brown LL, Schneider JS, and Lidsky TI. Sensory and cognitive functions of the basal ganglia. Curr Opin Neurobiol 7: 157–163, 1997.[CrossRef][Web of Science][Medline]
Brown RG and Pluck G. Negative symptoms: the "pathology " of motivation and goal-directed behaviour. Trends Neurosci 23: 412–417, 2000.[CrossRef][Web of Science][Medline]
Calabresi P, Maj R, Pisani A, Mercuri NB, and Bernardi G. Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J Neurosci 12: 4224–4233, 1992.[Abstract]
Calabresi P, Pisani A, Mercuri NB, and Bernardi G. The corticostriatal projection: from synaptic plasticity to dysfunctions of the basal ganglia. Trends Neurosci 19: 19–24, 1996.[CrossRef][Web of Science][Medline]
Carr GD and White NM. Conditioned place preference from intra-accumbens but not intra-caudate amphetamine injections. Life Sci 33: 2551–2557, 1983.[CrossRef][Web of Science][Medline]
Charpier S and Deniau JM. In vivo activity-dependent plasticity at cortico-striatal connections: evidence for physiological long-term potentiation. Proc Natl Acad Sci USA 94: 7036–7040, 1997.
Chevalier G and Deniau JM. Disinhibition as a basic process in the expression of striatal functions. Trends Neurosci 13: 277–280, 1990.[CrossRef][Web of Science][Medline]
Chevalier G, Vacher S, and Deniau JM. Inhibitory nigral influence on tectospinal neurons, a possible implication of basal ganglia in orienting behavior. Exp Brain Res 53: 320–326, 1984.[Web of Science][Medline]
Chevalier G, Vacher S, Deniau JM, and Desban M. Disinhibition as a basic process in the expression of striatal functions. I. The striato-nigral influence on tecto-spino/tecto-diencephalic neurons. Brain Res 334: 215–226, 1985.[CrossRef][Web of Science][Medline]
Chiara GD, Morelli M, and Consolo S. Modulatory functions of neurotransmitters in the striatum: ACh/dopamine/NMDA interactions. Trends Neurosci 17: 228–233, 1994.[CrossRef][Web of Science][Medline]
Christoph GR, Leonzio RJ, and Wilcox KS. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. J Neurosci 6: 613–619, 1986.[Abstract]
Corbett D and Wise RA. Intracranial self-stimulation in relation to the ascending dopaminergic systems of the midbrain: a moveable electrode mapping study. Brain Res 185: 1–15, 1980.[CrossRef][Web of Science][Medline]
Cromwell HC and Schultz W. Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. J Neurophysiol 89: 2823–2838, 2003.
Dayan P and Balleine B. Reward, motivation, and reinforcement learning. Neuron 36: 285–298, 2002.[CrossRef][Web of Science][Medline]
Delgado MR, Nystrom LE, Fissell C, Noll DC, and Fiez JA. Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84: 3072–3077, 2000.
DeLong MR and Georgopoulos AP. Motor functions of the basal ganglia. In: Handbook of Physiology. The Nervous System. Motor Control. Bethesda, MD: Am. Physiol. Soc., 1981, sect. 1, vol. II, pt. 2, p. 1017–1061.
Denny-Brown D. The Basal Ganglia and Their Relation to Disorders of Movement. London: Oxford Univ. Press, 1962.
Denny-Brown D. Clinical symptomatology of diseases of the basal ganglia. In: Diseases of the Basal Ganglia, edited by Vinken PJ and Bruyn GW. Amsterdam: North-Holland, 1968, p. 133–172.
Dommett E, Coizet V, Blaha CD, Martindale J, Lefebvre V, Walton N, Mayhew JE, Overton PG, and Redgrave P. How visual stimuli activate dopaminergic neurons at short latency. Science 307: 1476–1479, 2005.
Elliott R, Friston KJ, and Dolan RJ. Dissociable neural responses in human reward systems. J Neurosci 20: 6159–6165, 2000.
Everitt BJ, Morris KA, O'Brien A, and Robbins TW. The basolateral amygdala-ventral striatal system and conditioned place preference: further evidence of limbic-striatal interactions underlying reward-related process. Neuroscience 42: 1–18, 1991.[CrossRef][Web of Science][Medline]
Everitt BJ, Parkinson JA, Olmstead MC, Arroyo M, Robledo P, and Robbins TW. Associative processes in addiction and reward. The role of amygdala-ventral striatal subsystems. Ann NY Acad Sci 877: 412–438, 1999.[CrossRef][Web of Science][Medline]
Ewert J-P. Neuroethology. Berlin: Springer-Verlag, 1980.
Fadel J and Deutch AY. Anatomical substrates of orexin-dopamine interactions: lateral hypothalamic projections to the ventral tegmental area. Neuroscience 111: 379–387, 2002.[CrossRef][Web of Science][Medline]
Flaherty AW and Graybiel AM. Motor and somatosensory corticostriatal projection magnifications in the squirrel monkey. J Neurophysiol 74: 2638–2648, 1995.
Fonnum F, Gottesfeld Z, and Grofova I. Distribution of glutamate decarboxylase, choline acetyltransferase and aromatic amino acid decarboxylase in the basal ganglia of normal and operated rats. Evidence for striatopallidal, striatopeduncular and striatonigral GABAergic fibres. Brain Res 143: 125–138, 1978.[CrossRef][Web of Science][Medline]
François C, Yelnik J, Percheron G, and Fénelon G. Topographic distribution of the axonal endings from the sensorimotor and associative striatum in the macaque pallidum and substantia nigra. Exp Brain Res 102: 305–318, 1994.[Web of Science][Medline]
Fudge JL and Haber SN. The central nucleus of the amygdala projection to dopamine subpopulations in primates. Neuroscience 97: 479–494, 2000.[CrossRef][Web of Science][Medline]
Fudge JL and Haber SN. Bed nucleus of the stria terminalis and extended amygdala inputs to dopamine subpopulations in primates. Neuroscience 104: 807–827, 2001.[CrossRef][Web of Science][Medline]
Fudge JL, Kunishio K, Walsh P, Richard C, and Haber SN. Amygdaloid projections to ventromedial striatal subterritories in the primate. Neuroscience 110: 257–275, 2002.[CrossRef][Web of Science][Medline]
Funahashi S, Bruce CJ, and Goldman-Rakic PS. Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J Neurophysiol 61: 331–349, 1989.
Georges F and Aston-Jones G. Potent regulation of midbrain dopamine neurons by the bed nucleus of the stria terminalis. J Neurosci 21(RC160): 161–166, 2001.
Gerfen CR, Engber TM, Mahan LC, Susel Z, Chase TN, Monsma JFJ, and Sibley DR. D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science 250: 1429–1432, 1990.
Glimcher PW. Making choices: the neurophysiology of visual-saccadic decision making. Trends Neurosci 4: 654–659, 2001.
Glimcher PW and Rustichini A. Neuroeconomics: the consilience of brain and decision. Science 306: 447–452, 2004.
Grantyn A, Grantyn R, Robiné K-P, and Berthoz A. Electroanatomy of tectal efferent connections related to eye movements in the horizontal plane. Exp Brain Res 37: 149–172, 1979.[CrossRef][Web of Science][Medline]
Graybiel AM. Organization of the nigrotectal connection: an experimental tracer study in the cat. Brain Res 143: 339–348, 1978.[CrossRef][Web of Science][Medline]
Graybiel AM. Neurotransmitters and neuromodulators in the basal ganglia. Trends Neurosci 13: 244–254, 1990.[CrossRef][Web of Science][Medline]
Graybiel AM. The basal ganglia and chunking of action repertoires. Neurobiol Learn Mem 70: 119–136, 1998.[CrossRef][Web of Science][Medline]
Grillner S, Hellgren J, Menard A, Saitoh K, and Wikstrom MA. Mechanisms for selection of basic motor programs—roles for the striatum and pallidum. Trends Neurosci 28: 364–370, 2005.[CrossRef][Web of Science][Medline]
Haber SN, Fudge JL, and McFarland NR. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20: 2369–2382, 2000.
Haber SN, Kunishio K, Mizobuchi M, and Lynd-Balta E. The orbital and medial prefrontal circuit through the primate basal ganglia. J Neurosci 15: 4851–4867, 1995.[Abstract]
Haber SN and McFarland NR. The concept of the ventral striatum in nonhuman primates. Ann NY Acad Sci 877: 33–48, 1999.[CrossRef][Web of Science][Medline]
Haruno M, Kuroda T, Doya K, Toyama K, Kimura M, Samejima K, Imamizu H, and Kawato M. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J Neurosci 24: 1660–1665, 2004.
Hayhoe M and Ballard D. Eye movements in natural behavior. Trends Cogn Sci 9: 188–194, 2005.[CrossRef][Web of Science][Medline]
Hedreen JC and DeLong MR. Organization of striatopallidal, striatonigral, and nigrostriatal projections in the macaque. J Comp Neurol 304: 569–595, 1991.[CrossRef][Web of Science][Medline]
Hikosaka O, Matsumura M, Kojima J, and Gardiner TW. Role of basal ganglia in initiation and suppression of saccadic eye movements. In: Role of the Cerebellum and Basal Ganglia in Voluntary Movement, edited by Mano N, Hamada I, and DeLong MR. Amsterdam: Elsevier, 1993a, p. 213–219.
Hikosaka O, Sakamoto M, and Miyashita N. Effects of caudate nucleus stimulation on substantia nigra cell activity in monkey. Exp Brain Res 95: 457–472, 1993b.[Web of Science][Medline]
Hikosaka O, Sakamoto M, and Usui S. Functional properties of monkey caudate neurons. I. Activities related to saccadic eye movements. J Neurophysiol 61: 780–798, 1989a.
Hikosaka O, Sakamoto M, and Usui S. Functional properties of monkey caudate neurons. II. Visual and auditory responses. J Neurophysiol 61: 799–813, 1989b.
Hikosaka O, Sakamoto M, and Usui S. Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J Neurophysiol 61: 814–832, 1989c.
Hikosaka O, Takikawa Y, and Kawagoe R. Role of the basal ganglia in the control of purposive saccadic eye movements. Physiol Rev 80: 953–978, 2000.
Hikosaka O and Wurtz RH. Visual and oculomotor functions of monkey substantia nigra pars reticulata. III. Memory-contingent visual and saccade responses. J Neurophysiol 49: 1268–1284, 1983a.
Hikosaka O and Wurtz RH. Visual and oculomotor functions of monkey substantia nigra pars reticulata. IV. Relation of substantia nigra to superior colliculus. J Neurophysiol 49: 1285–1301, 1983b.
Hollerman JR and Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1: 304–309, 1998.[CrossRef][Web of Science][Medline]
Hollerman JR, Tremblay L, and Schultz W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J Neurophysiol 80: 947–963, 1998.
Hoover JE and Strick PL. The organization of cerebellar and basal ganglia outputs to primary motor cortex as revealed by retrograde transneuronal transport of herpes simplex virus type 1. J Neurosci 19: 1446–1463, 1999.
Hotson JR, Langston EB, and Langston JW. Saccade responses to dopamine in human MPTP-induced parkinsonism. Ann Neurol 20: 456–463, 1986.[CrossRef][Web of Science][Medline]
Houk JC. Information processing in modular circuits linking basal ganglia and cerebral cortex. In: Models of Information Processing in the Basal Ganglia, edited by Houk JC, Davis JL, and Beiser DG. Cambridge, MA: MIT Press, 1995, p. 3–9.
Houk JC, Adams JL, and Barto A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Models of Information Processing in the Basal Ganglia, edited by Houk JC, Davis JL, and Beiser DG. Cambridge, MA: MIT Press, 1995, p. 249–270.
Ikeda T and Hikosaka O. Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39: 693–700, 2003.[CrossRef][Web of Science][Medline]
Inoue M, Oomura Y, Aou S, Nishino H, and Sikdar SK. Reward related neuronal activity in monkey dorsolateral prefrontal cortex during feeding behavior. Brain Res 326: 307–312, 1985.[CrossRef][Web of Science][Medline]
Iribe Y, Moore K, Pang KCH, and Tepper JM. Subthalamic stimulation-induced synaptic responses in substantia nigra pars compacta dopaminergic neurons in vitro. J Neurophysiol 82: 925–933, 1999.
Itoh H, Nakahara H, Hikosaka O, Kawagoe R, Takikawa Y, and Aihara K. Correlation of primate caudate neural activity and saccade parameters in reward-oriented behavior. J Neurophysiol 89: 1774–1783, 2003.
Jayaraman A, Batton RRI, and Carpenter MB. Nigrotectal projections in the monkey: an autoradiographic study. Brain Res 135: 147–152, 1977.[CrossRef][Web of Science][Medline]
Johansson RS, Westling G, Bäckström A, and Flanagan JR. Eye-hand coordination in object manipulation. J Neurosci 21: 6917–6932, 2001.
Joseph JP and Boussaoud D. Role of the cat substantia nigra pars reticulata in eye and head movements. I. Neural activity. Exp Brain Res 57: 286–296, 1985.[Web of Science][Medline]
Karabelas AB and Moschovakis AK. Nigral inhibitory termination on efferent neurons of the superior colliculus: an intracellular horseradish peroxidase study in the cat. J Comp Neurol 239: 309–329, 1985.[CrossRef][Web of Science][Medline]
Kato M, Miyashita N, Hikosaka O, Matsumura M, Usui S, and Kori A. Eye movements in monkeys with local dopamine depletion in the caudate nucleus. I. Deficits in spontaneous saccades. J Neurosci 15: 912–927, 1995.[Abstract]
Kawagoe R, Takikawa Y, and Hikosaka O. Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci 1: 411–416, 1998.[CrossRef][Web of Science][Medline]
Kawagoe R, Takikawa Y, and Hikosaka O. Reward-predicting activity of dopamine and caudate neurons: a possible mechanism of motivational control of saccadic eye movement. J Neurophysiol 91: 1013–1024, 2004.
Kelley AE. Ventral striatal control of appetitive motivation: role in ingestive behavior and reward-related learning. Neurosci Biobehav Rev 27: 765–776, 2004.[CrossRef][Web of Science][Medline]
Kimura M, Kato M, and Shimazaki H. Physiological properties of projection neurons in the monkey striatum to the globus pallidus. Exp Brain Res 82: 672–676, 1990.[Web of Science][Medline]
Kita H. GABAergic circuits of the striatum. In: Chemical Signalling in the Basal Ganglia, edited by Arbuthnott GW and Emson PC. Amsterdam: Elsevier, 1993, p. 51–72.
Knutson B, Adams CM, Fong GW, and Hommer D. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J Neurosci 21(RC159): 151–155, 2001.
Kobayashi S, Lauwereyns J, Koizumi M, Sakagami M, and Hikosaka O. Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. J Neurophysiol 87: 1488–1498, 2002a.
Kobayashi Y, Inoue Y, Yamamoto M, Isa T, and Aizawa H. Contribution of pedunculopontine tegmental nucleus neurons to performance of visually guided saccade tasks in monkeys. J Neurophysiol 88: 715–731, 2002b.
Künzle H. Bilateral projections from the precentral motor cortex to the putamen and other parts of the basal ganglia. An autoradiographic study in Macaca fascicularis. Brain Res 88: 195–209, 1975.[CrossRef][Web of Science][Medline]
Lauwereyns J, Takikawa Y, Kawagoe R, Kobayashi S, Koizumi M, Coe B, Sakagami M, and Hikosaka O. Feature-based anticipation of cues that predict reward in monkey caudate nucleus. Neuron 33: 463–473, 2002a.[CrossRef][Web of Science][Medline]
Lauwereyns J, Watanabe K, Coe B, and Hikosaka O. A neural correlate of response bias in monkey caudate nucleus. Nature 418: 413–417, 2002b.[CrossRef][Medline]
Lavoie B and Parent A. Pedunculopontine nucleus in the squirrel monkey: cholinergic and glutamatergic projections to the substantia nigra. J Comp Neurol 344: 232–241, 1994.[CrossRef][Web of Science][Medline]
Lee HJ, Groshek F, Petrovich GD, Cantalini JP, Gallagher M, and Holland PC. Role of amygdalo-nigral circuitry in conditioning of a visual stimulus paired with food. J Neurosci 25: 3881–3888, 2005.
Leon MI and Shadlen MN. Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron 24: 415–425, 1999.[CrossRef][Web of Science][Medline]
Liu Z and Richmond BJ. Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules. J Neurophysiol 83: 1677–1692, 2000.
Lovinger DM, Partridge JG, and Tang KC. Plastic control of striatal glutamatergic transmission by ensemble actions of several neurotransmitters and targets for drugs of abuse. Ann NY Acad Sci 1003: 226–240, 2003.[CrossRef][Web of Science][Medline]
Lynd-Balta E and Haber SN. The organization of midbrain projections to the striatum in the primate: sensorimotor-related striatum versus ventral striatum. Neuroscience 59: 625–640, 1994a.[CrossRef][Web of Science][Medline]
Lynd-Balta E and Haber SN. Primate striatonigral projections: a comparison of the sensorimotor-related striatum and the ventral striatum. J Comp Neurol 345: 562–578, 1994b.[CrossRef][Web of Science][Medline]
Mahon S, Deniau JM, and Charpier S. Corticostriatal plasticity: life after the depression. Trends Neurosci 27: 460–467, 2004.[CrossRef][Web of Science][Medline]
Maunsell JH. Neuronal representations of cognitive state: reward or attention? Trends Cogn Sci 8: 261–265, 2004.[CrossRef][Web of Science][Medline]
May PJ and Hall WC. Relationships between the nigrotectal pathway and the cells of origin of the predorsal bundle. J Comp Neurol 226: 357–376, 1984.[CrossRef][Web of Science][Medline]
McClure SM, Laibson DI, Loewenstein G, and Cohen JD. Separate neural systems value immediate and delayed monetary rewards. Science 306: 503–507, 2004a.
McClure SM, York MK, and Montague PR. The neural substrates of reward processing in humans: the modern role of fMRI. Neuroscientist 10: 260–268, 2004b.
Middleton FA and Strick PL. Anatomical evidence for cerebellar and basal ganglia involvement in higher cognitive functions. Science 266: 458–461, 1994.
Middleton FA and Strick PL. Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Res Rev 31: 236–250, 2000.[CrossRef][Medline]
Minamimoto T, Hori Y, and Kimura M. Complementary process to response bias in the centromedian nucleus of the thalamus. Science 308: 1798–1801, 2005.
Mink JW. The basal ganglia: focused selection and inhibition of competing motor programs. Prog Neurobiol 50: 381–425, 1996.[CrossRef][Web of Science][Medline]
Mirenowicz J and Schultz W. Importance of unpredictability for reward responses in primate dopamine neurons. J Neurophysiol 72: 1024–1027, 1994.
Miyashita K, Rand MK, Miyachi S, and Hikosaka O. Anticipatory saccades in sequential procedural learning in monkeys. J Neurophysiol 76: 1361–1366, 1996.
Miyashita N, Hikosaka O, and Kato M. Visual hemineglect induced by unilateral striatal dopamine dificiency in monkeys. Neuroreport 6: 1257–1260, 1995.[Web of Science][Medline]
Miyazaki K, Mogi E, Araki N, and Matsumoto G. Reward-quality dependent anticipation in rat nucleus accumbens. Neuroreport 9: 3943–3948, 1998.[Web of Science][Medline]
Mogenson GJ, Jones DL, and Yim CY. From motivation to action: functional interface between the limbic system and the motor system. Prog Neurobiol 14: 69–97, 1980.[CrossRef][Web of Science][Medline]
Montague P and Berns G. Neural economics and the biological substrates of valuation. Neuron 36: 265–284, 2002.[CrossRef][Web of Science][Medline]
Montague PR, Dayan P, and Sejnowski TJ. A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16: 1936–1947, 1996.
Montague PR, Hyman SE, and Cohen JD. Computational roles for dopamine in behavioural control. Nature 431: 760–767, 2004.[CrossRef][Medline]
Mora F, Rolls ET, Burton MJ, and Shaw GS. Effects of dopamine-receptor blockade on self-stimulation in the monkey. Pharmacol Biochem Behav 4: 211–216, 1976.[CrossRef][Web of Science][Medline]
Nakahara H, Amari S, and Hikosaka O. Self-organization in the basal ganglia with modulation of reinforcement signals. Neural Comput 14: 819–844, 2002.[CrossRef][Web of Science][Medline]
Nakahara H, Doya K, and Hikosaka O. Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences—a computational approach. J Cogn Neurosci 13: 626–647, 2001.[CrossRef][Web of Science][Medline]
Nakahara H, Itoh H, Kawagoe R, Takikawa Y, and Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 41: 269–280, 2004.[CrossRef][Web of Science][Medline]
Nakamura K and Hikosaka O. Electrical stimulation in the primate caudate nucleus reinforces saccadic eye movement. 2003 Abstract Viewer and Itinerary Planner. Washington, DC: Society for Neuroscience, Online, 2003.
Nakamura K and Hikosaka O. Reward-dependent saccade bias is attenuated by local application of dopamine antagonist in the primate caudate nucleus. 2004 Abstract Viewer and Itinerary Planner. Washington, DC: Society for Neuroscience, Online, 2004.
Nakano K, Hasegawa Y, Tokushige A, Nakagawa S, Kayahara T, and Mizuno N. Topographical projections from the thalamus, subthalamic nucleus and pedunculopontine tegmental nucleus to the striatum in the japanese monkey, Macaca fuscata. Brain Res 537: 54–68, 1990.[CrossRef][Web of Science][Medline]
Nambu A, Kaneda K, Tokuno H, and Takada M. Organization of corticostriatal motor inputs in monkey putamen. J Neurophysiol 88: 1830–1842, 2002.
Nicola SM, Surmeier DJ, and Malenka RC. Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu Rev Neurosci 23: 185–215, 2000.[CrossRef][Web of Science][Medline]
Nicola SM, Yun IA, Wakabayashi KT, and Fields HL. Cue-evoked firing of nucleus accumbens neurons encodes motivational significance during a discriminative stimulus task. J Neurophysiol 91: 1840–1865, 2004.
Nishino H, Ono T, Sasaki K, Fukuda M, and Muramoto K-I. Caudate unit activity during operant feeding behavior in monkeys and modulation by cooling prefrontal cortex. Behav Brain Res 11: 21–33, 1984.[CrossRef][Web of Science][Medline]
O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, and Dolan RJ. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: 452–454, 2004.
O'Doherty JP, Deichmann R, Critchley HD, and Dolan RJ. Neural responses during anticipation of a primary taste reward. Neuron 33: 815–826, 2002.[CrossRef][Web of Science][Medline]
Olds J and Milner P. Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. J Comp Physiol Psychol 47: 419–427, 1954.[Medline]
Packard MG and Knowlton BJ. Learning and memory functions of the basal ganglia. Annu Rev Neurosci 25: 563–593, 2002.[CrossRef][Web of Science][Medline]
Pan WX and Hyland BI. Pedunculopontine tegmental nucleus controls conditioned responses of midbrain dopamine neurons in behaving rats. J Neurosci 25: 4725–4732, 2005.
Paré M and Wurtz RH. Progression in neuronal processing for saccadic eye movements from parietal cortex area LIP to superior colliculus. J Neurophysiol 85: 2545–2562, 2001.
Parent A and Hazrati L-N. Multiple striatal representation in primate substantia nigra. J Comp Neurol 344: 305–320, 1994.[CrossRef][Web of Science][Medline]
Parent A and Hazrati L-N. Functional anatomy of the basal ganglia. I. The cortico-basal ganglia-thalamo-cortical loop. Brain Res Rev 20: 91–127, 1995.[CrossRef][Medline]
Parent A, Mackey A, and De Bellefeuille L. The subcortical afferents to caudate nucleus and putamen in primate: a fluorescence retrograde double labeling study. Neuroscience 10: 1137–1150, 1983.[CrossRef][Web of Science][Medline]
Parthasarathy HB, Schall JD, and Graybiel AM. Distributed but convergent ordering of corticostriatal projections: analysis of the frontal eye field and the supplementary eye field in the macaque monkey. J Neurosci 12: 4468–4488, 1992.[Abstract]
Pasupathy A and Miller EK. Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433: 873–876, 2005.[CrossRef][Medline]
Phillips AG, Carter DA, and Fibiger HC. Dopaminergic substrates of intracranial self-stimulation in the caudate-putamen. Brain Res 104: 221–232, 1976.[CrossRef][Web of Science][Medline]
Platt ML and Glimcher PW. Neural correlates of decision variables in parietal cortex. Nature 400: 233–238, 1999.[CrossRef][Medline]
Preston RJ, Bishop GA, and Kitai ST. Medium spiny neuron projection from the rat striatum: an intracellular horseradish peroxidase study. Brain Res 183: 253–263, 1980.[Web of Science][Medline]
Ragsdale CW and Graybiel AM. Fibers from the basolateral nucleus of the amygdala selectively innervate striosomes in the caudate nucleus of the cat. J Comp Neurol 269: 506–522, 1988.[CrossRef][Web of Science][Medline]
Rescorla RA and Wagner AR. A theory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical Conditioning II: Current Research and Theory, edited by Black AH and Prokasy WF. New York: Appleton-Century-Croft, 1972, p. 64–99.
Reynolds JN and Wickens JR. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15: 507–521, 2002.[CrossRef][Web of Science][Medline]
Reynolds JNJ and Wickens JR. Substantia nigra dopamine regulates synaptic plasticity and membrane potential fluctuations in the rat neostriatum, in vivo. Neuroscience 99: 199–203, 2000.[CrossRef][Web of Science][Medline]
Reynolds JNR, Hyland BI, and Wickens JR. A cellular mechanism of reward-related learning. Nature 413: 67–70, 2001.[CrossRef][Medline]
Roesch MR and Olson CR. Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. J Neurophysiol 90: 1766–1789, 2003.
Roesch MR and Olson CR. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304: 307–310, 2004.
Rolls ET, Thorpe SJ, and Maddison SP. Responses of striatal neurons in the behaving monkey. I. Head of the caudate nucleus. Behav Brain Res 7: 179–210, 1983.[CrossRef][Web of Science][Medline]
Sachs OW. Awakenings. New York: Dutton, 1983.
Sadikot AF, Parent A, and François C. Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a PHA-L study of subcortical projections. J Comp Neurol 315: 137–159, 1992.[CrossRef][Web of Science][Medline]
Salmon DP and Butters N. Neurobiology of skill and habit learning. Curr Opin Neurobiol 5: 184–190, 1995.[CrossRef]
Sato M and Hikosaka O. Role of primate substantia nigra pars reticulata in reward-oriented saccadic eye movement. J Neurosci 22: 2363–2373, 2002.
Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1–27, 1998.
Schultz W, Apicella P, and Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13: 900–913, 1993.[Abstract]
Schultz W, Apicella P, Scarnati E, and Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci 12: 4595–4610, 1992.[Abstract]
Schultz W, Dayan P, and Montague PR. A neural substrate of prediction and reward. Science 275: 1593–1599, 1997.
Schultz W and Dickinson A. Neuronal coding of prediction errors. Annu Rev Neurosci 23: 473–500, 2000.[CrossRef][Web of Science][Medline]
Segraves MA and Goldberg ME. Functional properties of corticotectal neurons in the monkey's frontal eye field. J Neurophysiol 58: 1387–1419, 1987.
Selby G. Parkinson's Disease, edited by Vinken PJ and Bruyn GW. Amsterdam: North-Holland, 1968, p. 173–211.
Selemon LD and Goldman-Rakic PS. Longitudinal topography and interdigitation of corticostriatal projections in the rhesus monkey. J Neurosci 5: 776–794, 1985.[Abstract]
Setlow B, Schoenbaum G, and Gallagher M. Neural encoding in ventral striatum during olfactory discrimination learning. Neuron 38: 625–636, 2003.[CrossRef][Web of Science][Medline]
Shidara M, Aigner TG, and Richmond BJ. Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J Neurosci 18: 2613–2625, 1998.
Shook BL, Schlag-Rey M, and Schlag J. Primate supplementary eye field. II. Comparative aspects of connections with the thalamus, corpus striatum, and related forebrain nuclei. J Comp Neurol 307: 562–583, 1991.[CrossRef][Web of Science][Medline]
Smith Y, Bevan MD, Shink E, and Bolam JP. Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 86: 353–387, 1998.[CrossRef][Web of Science][Medline]
Sommer MA and Wurtz RH. Composition and topographic organization of signals sent from the frontal eye field to the superior colliculus. J Neurophysiol 83: 1979–2001, 2000.
Sparks DL. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol Rev 66: 118–171, 1986.
Sparks DL. The brainstem control of saccadic eye movements. Nat Rev Neurosci 3: 952–964, 2002.[CrossRef][Web of Science][Medline]
Sparks DL, Holland R, and Guthrie BL. Size and distribution of movement fields in the monkey superior colliculus. Brain Res 113: 21–34, 1976.[CrossRef][Web of Science][Medline]
Spyraki C, Fibiger HC, and Phillips AG. Dopaminergic substrates of amphetamine-induced place preference conditioning. Brain Res 253: 185–193, 1982.[CrossRef][Web of Science][Medline]
Stanton GB, Goldberg ME, and Bruce CJ. Frontal eye field efferents in the macaque monkey: I. Subcortical pathways and topography of striatal and thalamic terminal fields. J Comp Neurol 271: 473–492, 1988.[CrossRef][Web of Science][Medline]
Sugrue LP, Corrado GS, and Newsome WT. Matching behavior and the representation of value in the parietal cortex. Science 304: 1782–1787, 2004.
Sugrue LP, Corrado GS, and Newsome WT. Choosing the greater of two goods: neural currencies for valuation and decision making. Nat Rev Neurosci 6: 363–375, 2005.[CrossRef][Web of Science][Medline]
Suri RE and Schultz W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res 121: 350–354, 1998.[CrossRef][Web of Science][Medline]
Surmeier DJ, Song W-J, and Yan Z. Coordinated expression of dopamine receptors in neostriatal medium spiny neurons. J Neurosci 16: 6579–6591, 1996.
Sutton RS and Barto AG. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press/Bradford, 1998.
Swanson LW. Cerebral hemisphere regulation of motivated behavior. Brain Res 886: 113–164, 2000.[CrossRef][Web of Science][Medline]
Takada M, Itoh K, Yasui Y, Sugimoto T, and Mizuno N. Topographical projections from the posterior thalamic regions to the striatum in the cat, with reference to possible tecto-thalamo-striatal connections. Exp Brain Res 60: 385–396, 1985.[Web of Science][Medline]
Takada M, Tokuno H, Nambu A, and Inase M. Corticostriatal projections from the somatic motor areas of the frontal cortex in the macaque monkey: segregation versus overlap of input zones from the primary motor cortex, the supplementary motor area, and the premotor cortex. Exp Brain Res 120: 114–128, 1998.[CrossRef][Web of Science][Medline]
Takakusaki K, Saitoh K, Harada H, and Kashiwayanagi M. Role of basal ganglia-brainstem pathways in the control of motor behaviors. Neurosci Res 50: 137–151, 2004.[CrossRef][Web of Science][Medline]
Takikawa Y, Kawagoe R, and Hikosaka O. Reward-dependent spatial selectivity of anticipatory activity in monkey caudate neurons. J Neurophysiol 87: 508–515, 2002a.
Takikawa Y, Kawagoe R, and Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. J Neurophysiol 92: 2520–2529, 2004.
Takikawa Y, Kawagoe R, Itoh H, Nakahara H, and Hikosaka O. Modulation of saccadic eye movements by predicted reward outcome. Exp Brain Res 142: 284–291, 2002b.[CrossRef][Web of Science][Medline]
Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, and Yamawaki S. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7: 887–893, 2004.[CrossRef][Web of Science][Medline]
Tepper JM, Martin LP, and Anderson DR. GABAA receptor-mediated inhibition of rat substantia nigra dopaminergic neurons by pars reticulata projection neurons. J Neurosci 15: 3092–3103, 1995.[Abstract]
Tobler PN, Dickinson A, and Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J Neurosci 23: 10402–10410, 2003.
Tobler PN, Fiorillo CD, and Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307: 1642–1645, 2005.
Tremblay L, Hollerman JR, and Schultz W. Modifications of reward expectation-related neuronal activity during learning in primate striatum. J Neurophysiol 80: 964–977, 1998.
Tricomi EM, Delgado MR, and Fiez JA. Modulation of caudate activity by action contingency. Neuron 41: 281–292, 2004.[CrossRef][Web of Science][Medline]
Triesch J, Ballard DH, Hayhoe MM, and Sullivan BT. What you see is what you need. J Vis 3: 86–94, 2003.[CrossRef][Web of Science][Medline]
Tulloch IF, Arbuthnott GW, and Wright AK. Topographical organization of the striatonigral pathway revealed by anterograde and retrograde neuroanatomical tracing techniques. J Anat 127: 425–441, 1978.[Web of Science][Medline]
Ullsperger M and von Cramon DY. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J Neurosci 23: 4308–4314, 2003.
Uno M and Ozawa N. Long-term potentiation of the amygdalo-striatal synaptic transmission in the course of development of amygdaloid kindling in cats. Neurosci Res 12: 251–262, 1991.[CrossRef][Web of Science][Medline]
Van den Pol AN, Smith AD, and Powell JF. GABA axons in synaptic contact with dopamine neurons in the substantia nigra: double immunocytochemistry with biotin-peroxidase and protein A-colloidal gold. Brain Res 348: 146–154, 1985.[CrossRef][Web of Science][Medline]
Waelti P, Dickinson A, and Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412: 43–48, 2001.[CrossRef][Medline]
Watanabe K and Hikosaka O. Immediate changes in anticipatory activity of caudate neurons associated with reversal of position-reward contingency. J Neurophysiol 94: 1879–1887, 2005.
Watanabe K, Lauwereyns J, and Hikosaka O. Neural correlates of rewarded and unrewarded eye movements in the primate caudate nucleus. J Neurosci 23: 10052–10057, 2003.
Watanabe M. Reward expectancy in primate prefrontal neurons. Nature 382: 629–632, 1996.[CrossRef][Medline]
Watanabe M, Hikosaka K, Sakagami M, and Shirakawa S. Coding and monitoring of motivational context in the primate prefrontal cortex. J Neurosci 22: 2391–2400, 2002.
Wickens J and Kötter R. Cellular models of reinforcement. In: Models of Information Processing in the Basal Ganglia, edited by Houk JC, Davis JL, and Beiser DG. Cambridge, MA: MIT Press, 1995, p. 187–214.
Williams GV and Goldman-Rakic PS. Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature 376: 572–575, 1995.[CrossRef][Medline]
Wilson CJ and Kawaguchi Y. The origins of two-state spontaneous membrane potential fluctuations of neostriatal spiny neurons. J Neurosci 16: 2397–2410, 1996.
Wise RA. Neurobiology of addiction. Curr Opin Neurobiol 6: 243–251, 1996.[CrossRef][Web of Science][Medline]
Wurtz RH and Goldberg ME. Activity of superior colliculus in behaving monkey. III. Cells discharging before eye movements. J Neurophysiol 35: 575–586, 1972.
Wurtz RH, Sommer MA, Paré M, and Ferraina S. Signal transformations from cerebral cortex to superior colliculus for the generation of saccades. Vision Res 41: 3399–3412, 2001.[CrossRef][Web of Science][Medline]
Yeterian EH and Pandya DN. Prefrontostriatal connections in relation to cortical architectonic organization in rhesus monkeys. J Comp Neurol 312: 43–67, 1991.[CrossRef][Web of Science][Medline]
Yoshida M and Precht W. Monosynaptic inhibition of neurons in the substantia nigra by caudate-nigral fibers. Brain Res 32: 225–228, 1971.[CrossRef][Web of Science][Medline]
Zheng T and Wilson CJ. Corticostriatal combinatorics: the implications of corticostriatal axonal arborizations. J Neurophysiol 87: 1007–1017, 2002.
This article has been cited by other articles:
|
|
 |
|
  C.F. Geier, R. Terwilliger, T. Teslovich, K. Velanova, and B. Luna Immaturities in Reward Processing and Its Influence on Inhibitory Control in Adolescence Cereb Cortex, October 29, 2009; (2009) bhp225v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Jovancevic-Misic and M. Hayhoe Adaptive Gaze Control in Natural Environments J. Neurosci., May 13, 2009; 29(19): 6234 - 6238. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K.-i. Okada, K. Toyama, Y. Inoue, T. Isa, and Y. Kobayashi Different Pedunculopontine Tegmental Neurons Signal Predicted and Actual Task Rewards J. Neurosci., April 15, 2009; 29(15): 4858 - 4870. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. H. Kao, B. D. Wright, and A. J. Doupe Neurons in a Forebrain Nucleus Required for Vocal Plasticity Rapidly Switch between Precise Firing and Variable Bursting Depending on Social Context J. Neurosci., December 3, 2008; 28(49): 13232 - 13247. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. F. Atherton, D. L. Wokosin, S. Ramanathan, and M. D. Bevan Autonomous initiation and propagation of action potentials in neurons of the subthalamic nucleus J. Physiol., December 1, 2008; 586(23): 5679 - 5700. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
O. Hikosaka, S. R. Sesack, L. Lecourtier, and P. D. Shepard Habenula: Crossroad between the Basal Ganglia and the Limbic System J. Neurosci., November 12, 2008; 28(46): 11825 - 11829. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Kaneda, K. Isa, Y. Yanagawa, and T. Isa Nigral Inhibition of GABAergic Neurons in Mouse Superior Colliculus J. Neurosci., October 22, 2008; 28(43): 11071 - 11078. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Isoda and O. Hikosaka A Neural Correlate of Motivational Conflict in the Superior Colliculus of the Macaque J Neurophysiol, September 1, 2008; 100(3): 1332 - 1342. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Liu and M. A. Basso Substantia Nigra Stimulation Influences Monkey Superior Colliculus Neuronal Activity Bilaterally J Neurophysiol, August 1, 2008; 100(2): 1098 - 1112. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Nakamura, M. Matsumoto, and O. Hikosaka Reward-Dependent Modulation of Neuronal Activity in the Primate Dorsal Raphe Nucleus J. Neurosci., May 14, 2008; 28(20): 5331 - 5343. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. A. Farries and A. L. Fairhall Reinforcement Learning With Modulated Spike Timing Dependent Synaptic Plasticity J Neurophysiol, December 1, 2007; 98(6): 3648 - 3665. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. A. Taha, S. M. Nicola, and H. L. Fields Cue-evoked encoding of movement planning and execution in the rat nucleus accumbens J. Physiol., November 1, 2007; 584(3): 801 - 818. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Kojima and A. J. Doupe Song Selectivity in the Pallial-Basal Ganglia Song Circuit of Zebra Finches Raised Without Tutor Song Exposure J Neurophysiol, October 1, 2007; 98(4): 2099 - 2109. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Nevet, G. Morris, G. Saban, D. Arkadir, and H. Bergman Lack of Spike-Count and Spike-Time Correlations in the Substantia Nigra Reticulata Despite Overlap of Neural Responses J Neurophysiol, October 1, 2007; 98(4): 2232 - 2243. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. W. Balleine, M. R. Delgado, and O. Hikosaka The Role of the Dorsal Striatum in Reward and Decision-Making J. Neurosci., August 1, 2007; 27(31): 8161 - 8165. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Saitoh, A. Menard, and S. Grillner Tectal Control of Locomotion, Steering, and Eye Movements in Lamprey J Neurophysiol, April 1, 2007; 97(4): 3093 - 3108. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Ding and O. Hikosaka Temporal Development of Asymmetric Reward-Induced Bias in Macaques J Neurophysiol, January 1, 2007; 97(1): 57 - 61. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Nakamura and O. Hikosaka Facilitation of Saccadic Eye Movements by Postsaccadic Electrical Stimulation in the Primate Caudate J. Neurosci., December 13, 2006; 26(50): 12885 - 12895. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Soltani and X.-J. Wang A biophysically based neural model of matching law behavior: melioration by stochastic synapses. J. Neurosci., April 5, 2006; 26(14): 3731 - 3744. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| Visit Other APS Journals Online |
