INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Expectations for different magnitudes of reward have powerful effects on learning and performance (Black 1968; Campbell and Seiden 1974; Flaherty 1996). In discrete choice trials, animals consistently choose larger rewards compared with smaller rewards in both food-deprived and nondeprived conditions (Collier 1982; Tolman 1932). Chimpanzees trained in a choice task for rewards of varying magnitude with an important caveat that the chosen reward is given to the passive (observer) monkey while the remaining item goes to the selector, consistently selected the larger reward from the array of items, thereby receiving the smaller reward (Boysen and Berntson 2001). This inapt strategy suggests that an immediate, obvious magnitude difference in primary reward has a powerful overriding influence in directing behavioral reactions. More recently, it has been shown that details of behavioral reactions used to obtain different liquid rewards in a delayed response task varied dependently on the outcome predicted by the presentation of an instruction cue (Watanabe et al. 2001). Faster reaction times and longer anticipatory lick durations were seen in trials for the preferred outcome. These findings support the notion that animals vary details of their reactions within these classic delay tasks dependent on the estimated value of the future outcome.

This study was undertaken to search for the neural substrates involved in preparing the animals to make different behavioral reactions dependent on reward expectations. Candidate structures should be able to integrate information concerning the reward with the movement plans to produce appropriate responses according to expectations. One of the main neural structures involved in these types of motivational-sensorimotor integrations is the striatum (Berridge and Cromwell 1990; Mogenson et al. 1980; Robbins and Everitt 1992). There has been ample evidence that information concerning the reward influences striatal activity (Aosaki et al. 1994; Apicella et al. 1991; Bowman et al. 1996; Hikosaka et al. 1989b). This influence has been noted on several different types of striatal responses, including the activity at the time of reward reception following behavioral performance (Apicella et al. 1991) and an activation that precedes the reward delivery when the animal is expecting a particular reward outcome (Apicella et al. 1992; Schultz et al. 1992).

Recently, a series of papers (Hassani et al. 2001; Hollerman et al. 1998; Kawagoe et al. 1998; Watanabe 1996) has examined whether or not these activations in striatum or prefrontal cortex incorporate details concerning the upcoming reward. Differences between the rewarding outcomes were either reward versus no reward (Hollerman et al. 1998; Kawagoe et al. 1998) or a comparison between two different juice types (Hassani et al. 2001; Watanabe 1996). Results of these studies showed that not only did sustained activity that preceded the reward vary dependently on the reward about to be received, but also activity that was temporally separated from the reward, linked to the predictive instruction picture or to the motor response, itself, showed a dependence to the rewarding outcome to be received in the future. It was hypothesized that this influence from the upcoming reward on this activity contributed to expectation formation and was a critical component of the goal-directed behavior the animal committed prior to movement execution.

Here, we advance this theory a step farther by examining these activations while monkeys work for different reward magnitudes. By using reward magnitude differences, we take advantage of a precise parametric property in reward outcome that can be shifted by a specific value, and we can evaluate as to whether or not a similar shift is observed in the neural activity of striatal neurons. Magnitude shifts avoid confounding aspects of differing sensory qualities of the tastant in influencing neural activity. In addition, it places the neurophysiological findings in line with a large set of psychological studies that have used shifts in reward magnitude as the independent variable to examine the influence of reward contrast and incentive relativity on motivated behavior (Flaherty 1996). We used a modified version of the spatial delayed task to test, separately, the behavioral reaction to be completed (arm movement to the left vs. arm movement to the right) and the amount of liquid reward to be acquired (different magnitudes). These trial types allowed us to examine how expectations for different reward magnitudes could modulate neuronal activity during the decision, preparation, initiation, and execution of behavioral reactions. Furthermore, using this task allows us to directly compare our results with results from previous studies that have used similar procedures (Hassani et al. 2001; Hollerman et al. 1998). The results have been previously presented in abstract form (Cromwell and Schultz 2000).

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Subjects were two macaque monkeys (A: Macaca fascicularis, male, 2.8 kg; B: M. fascicularis, male, 2.5 kg). The activity of single cells was recorded using moveable microelectrodes during performance of a spatial delayed response task for different reward magnitudes. Arm muscle activity, licking movements, and eye movements were monitored during the task performance. Electrode positions were reconstructed from small electrolytic lesions on 50-µm-thick, cresyl violet-stained histological brain sections. Most methods were similar to those described in detail before (Hollerman et al. 1998). All experimental protocols conformed to National Institutes of Health Guidelines and the Swiss Animal Protection Law; they were supervised by the Fribourg Cantonal Veterinary Office.

Behavioral procedures

Animals performed in a spatial delayed response task for liquid reward using either their right or left hand. The monkey kept its hand relaxed on an immovable, touch-sensitive resting key. It faced a 13" computer monitor positioned behind a transparent plastic wall in which two small levers were mounted to the left and right of the midline. To start each trial, a color instruction picture (13° × 13°) appeared for 1 s on a computer screen above the left or right lever (Fig. 1; top). The instruction indicated both the target of a future arm movement by its position on the screen and the liquid reward delivered for correctly reacting to an upcoming trigger stimulus. After a randomly varying delay of 3.5-4.5 s following instruction onset, two identical red squares appeared simultaneously as movement trigger at the left and right positions of the instruction. The trigger determined the time of the behavioral response without indicating the spatial target or the specific reward. The animal released the resting key, touched the lever at the position previously indicated by the instruction, and received the liquid reward indicated by the instruction. Incorrect lever touch or failure to touch a lever went unrewarded. Both trigger squares extinguished on correct or incorrect lever touch or after 2.0 s without lever touch.

View larger version (47K): [in this window] [in a new window]   Fig. 1. Behavioral task and instruction picture sets. A: spatial delayed response task. An initial instruction picture of 1 s duration indicated the left or right movement target and the liquid reward amount delivered at trial end. After a random delay of 3.5-4.5 s following instruction onset, 2 identical squares appeared and elicited the movement from the resting key to the left or right target lever indicated by the instruction. Correct performance was rewarded after a delay of 2 s with 1 of 3 quantities of the liquid indicated by the previous instruction. B: most commonly used instruction pictures. Each instruction set contained 3 pictures for 3 different rewards shown inside vertical rectangles. Only 2 reward magnitudes with their corresponding associated pictures from a single picture set were used in a particular trial block.

A liquid reward drop was dispensed 2.0 s after lever touch by a computer-controlled liquid valve from a spout at the animal's mouth. Drop sizes were 0.12, 0.18, and 0.24 ml for the low, medium, and high magnitudes, respectively. Each reward amount was indicated at trial onset by a specific instruction picture. Different liquid types (black currant or raspberry juice for Animal A and grenadine or blackcurrant juice for Animal B) were used; however, only one liquid type was used within any one recording session. In this way, we did not confound the preferences for the different reward juices (Hassani et al. 2001) with the different reward amounts to be received. To assess the influence of visual features on neuronal responses, we used two to six different pairs of three instruction pictures in each animal (Fig. 1; bottom). Only two instruction pictures with their associated two liquid reward magnitudes were used in a given block of trials. All reward magnitudes were used in combinations in which animals showed reliable and persistent preferences (see Control Task below).

The two spatial targets and two liquid amounts alternated semi-randomly, with the consecutive occurrence of same trial types being restricted to three trials. Trials lasted 12 s irrespective of behavioral performance; intertrial intervals were 2-3 s. Closed-circuit video systems served to continuously supervise limb and mouth movements. Animals were partially fluid-deprived during weekdays and were returned to their home cages after each daily session.

Two additional trial types were used for control purposes. First, we assessed reward preferences in blocks of choice trials before or after recording from each neuron. Two different instructions for two reward magnitudes were shown simultaneously at semi-randomly alternating left and right target positions, allowing the animal to touch the lever of its choice following the trigger stimulus. Thus each pair of instruction stimuli contained one picture associated with the preferred liquid amount (larger magnitude) and one with the nonpreferred amount (smaller magnitude).

In the second task variation, we assessed further movement relationships of selected neurons in nonmovement trials that were semi-randomly interspersed with left and right movement trials in a delayed go-nogo task. The instruction picture was presented for 1.0 s at the center of the monitor instead of the left or right positions. The animal kept its hand on the resting key during the instruction-trigger delay and for 2.0 s during presentation of the same trigger stimulus as in movement trials to receive, after a further 2.0 s, the specific liquid reward magnitude indicated by the instruction. Thus nonmovement trials were indicated by the center position of the instruction.

Data acquisition

Following behavioral conditioning, animals were implanted with two horizontal cylinders for head fixation and a stainless steel chamber permitting vertical access with microelectrodes to the left striatum under deep sodium pentobarbital anesthesia and aseptic conditions. The dura was left intact. Teflon-coated, multistranded, stainless steel wires were implanted into the right extensor digitorum communis and biceps brachii muscles for EMG recordings. The implant was fixed to the skull with stainless steel screws and several layers of dental cement. Animals received postoperative antibiotics and analgesics.

Glass-insulated, platinum-plated tungsten microelectrodes positioned inside a metal guide cannula served to record extracellularly the activity of single neurons, using conventional electrophysiological techniques. Discharges from neuronal perikarya were converted into standard digital pulses by means of an adjustable Schmitt-trigger. EMGs and horizontal and vertical eye positions (infrared oculometer, Iscan) were collected during neuronal recordings. EMGs were converted into standard digital pulses by a Schmitt-trigger. Licking movements were recorded as standard digital pulses by tongue interruptions of an infrared lightbeam at the liquid spout.

Pulses from neuronal discharges and EMGs were sampled together with digital signals from the behavioral task by a computer, together with analog signals from EMGs using infrared monitoring of horizontal and vertical eye movements (Iscan). Only data from neurons sampled by the computer for 10 trials in each of the four trial types are reported. All data from neurons suspected to covary with some task component, and occasionally from unmodulated neurons, were stored on computer disks.

Data analysis

Data analysis was performed similarly to previous studies (see Hassani et al. 2001; Hollerman et al. 1998). Task-related neuronal changes were assessed during individual task periods with the nonparametric one-tailed Wilcoxon signed-rank test incorporated into the evaluation software (P < 0.01). Only task-related activations were considered from neurons, showing statistically significant activity increases in relation to at least one task event, compared with a 1- or 2-s control period that was either immediately before the instruction as first task event or at a period of apparent lack of modulation outside of the task sequence in cases of suspected preinstruction activations.

Task-related changes were compared in individual neurons during identical task periods and durations with the two-tailed Mann-Whitney U test (P < 0.01), separately between different magnitudes, between left and right targets, and between corresponding pictures of different instruction sets. Comparisons between these different conditions used the following standard time windows: 0-1 s after instruction, 0-3.0 s before the trigger, 0-1.0 s after trigger, 0-2.0 s before the reward, and 0-2.0 s after reward. Only data from neurons with insignificant differences in control periods were evaluated (P > 0.05). Reaction times (from trigger stimulus onset to key release) were collected during neuronal recordings. Because of these distributions being occasionally skewed, we compared reaction times with the nonparametric Wilcoxon matched pairs signed-rank test between trials with two levels of reward magnitude.

A simple linear regression was completed that took advantage of the parametric features of the reward magnitude comparisons. The relationship between spike rate and reward magnitude was analyzed using a linear regression model, y = bx + a, in which y is the normalized spike rate and x equals the different levels of reward magnitude. Reward magnitude was measured as a percentage of the highest liquid amount delivered (high level = 1, medium level = 0.7, and low level = 0.4). Normalization of spiking rate was completed only on the data used for the simple linear regression and was determined by dividing the average spike rate at the time of increased neural activity by the mean of the baseline spike rate. To compare between the different magnitude conditions, the baseline spike rate was obtained as the mean of the average firing rates across the different reward magnitude conditions for each neuron. The different conditions included low, medium, and high reward amount trials. Normalization was used to transform the data from neuronal activations tested using all three-reward magnitude levels in two different combinations (high level vs. medium level and medium level vs. low level) and not for neuronal activations tested in only two liquid magnitude conditions. Two levels of analysis were completed. One level was more conservative and included only those neurons that had a significant difference in task-related periods for the two reward magnitude comparisons completed (high level vs. medium level and medium level vs. low level), while the second level was less stringent and required that only one comparison be significantly different (high level vs. medium level or medium level vs. low level) and combined with the second comparison with a trend in the similar direction. The second analysis was completed due to the low neuron number obtained using the first criterion (n = 7 neurons), while the numbers of neurons were greater in the less stringent analysis (n = 39). The significance of the slope of the linear fit for the correlation coefficient r being different from zero was computed using an estimate of the SE, which is distributed as t on N  2 df:

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Behavioral performance

Behavioral data used to measure performance were obtained throughout the recording period. For both monkeys, this included 3 sessions from the first third, second third, and the last third of their recording periods. Within this sample, animals performed the task >95% correctly for all trial types. This level of competence as an average for all trial types shows that the animals learned the spatial response task well, and overall, performed at a high accuracy level. When accuracy was analyzed between different reward magnitude trial types, the performance level varied. Animals performed significantly more accurately in trials for large rewards (100%) versus trials for the smaller reward (91.5% median values; P < 0.001, Wilcoxon signed-rank test). In choice trials, where animals could choose either the larger or smaller of the reward liquid amounts, both subjects reliably chose the larger reward drop size (>95% of the trials). This preference was very consistent within a 2- to 3-h recording session. We chose trials to analyze behavioral reactions that had been preceded or followed by a choice test to better ensure that the animal was discriminating between the different reward magnitude conditions. In trials with a single target and a single reward, the animals showed greater anticipatory licking and shorter reaction times prior to receiving the larger reward amount and the inverse relationship in trials for the smaller reward drop size (Table 1). These differences in behavioral reactions were reliably consistent within the recording sessions for individual neurons and are very similar to behavioral results obtained in previous studies (Hassani et al. 2001; Watanabe et al. 2001). The behavioral reactions occur seconds prior to the delivery of the reward and suggest that the animal has formed an expectation for the particular reward magnitude by the time the behavioral reaction is initiated.

Muscle activity (extensor digitorium communis and biceps) was similar between trials for smaller or larger rewards (Fig. 2). Gaze and eye movements did not vary systematically between trials for larger or smaller rewards. Usually, the instruction cue elicited a saccade to the image, unless the gaze was already there. The trigger stimuli presentation elicited a saccade in most cases to the target lever to be pressed. We found no instances of activity during the delay period that could account for differences in the neural activity seen in the striatum similar to previous findings in very comparable task situations (Hassani et al. 2001; Tremblay and Schultz 1999).

View larger version (27K): [in this window] [in a new window]   Fig. 2. Activity of the biceps muscle of the right arm during spatial delayed response task performance. Trials alternated semirandomly between left and right target positions and between 2 liquid reward amounts indicated (Low Reward vs. High Reward). A: trials when the animal is reaching for the left lever. B: trials when the animal is reaching for the right lever. Activity did not differ between the trials for different reward magnitudes. Raster dots correspond to rectified activity above a threshold level. Individual trials are presented as horizontal lines and are ranked vertically according to instruction-trigger intervals.

Neuronal database

We studied 2,000 slowly discharging striatal neurons with control rates between 0.1 and 3 impulses/s in the spatial delayed response task. Of the 2,000 neurons, 374 showed 500 statistically significant task-related activations (190 and 184 in animals A and B, respectively). The number of activations is greater than the number of neurons because neurons could show more than one task-related event. The remaining cells failed to show task related modulations in raster displays during the experiment and were not further examined. Five different forms of task-related activity were uncovered during the trial periods and consisted of the following: 1) responses to instructions; 2) activations preceding trigger; 3) activations following the movement trigger; 4) activations preceding rewards; and 5) responses to the reward. The distinct group of tonically active neurons (Apicella et al. 1997; Kimura et al. 1984) with discharge rates of 3-8 impulses/s was not investigated.

A total of 204 of 374 task-related neurons (55%) showed statistically significant differences between at least two different reward magnitudes. Neurons could show more than one type of activation to an event or movement or preceding an event or movement so the total of activations (244) exceeds the total number of neurons (204). Activations were higher with either larger rewards (66% of discriminations; Figs. 5, 6, and 12) or smaller rewards (34%; Figs. 8, 10, 11, and 13) in all five types of task-related responses. In these same categories of responses (Instruction, Preceding-trigger, Following-trigger, Preceding-reward, and Reward), the proportion of neurons that had significantly different activity between different reward magnitude trials varied (27-100%: see Table 2). We determined the percent increase in activity from the baseline firing rate for the five main response types using the highest level of reward magnitude trials. Figure 3 presents the distribution of the activity changes for the different responses. The average percent increases of neural activity for the following response types were 121 ± 18% (SE) for the 61 instruction responses, 91 ± 12% for the 56 activations preceding the trigger, 69 ± 10% for the 31 activations following the trigger stimulus, 88 ± 13% for the 48 activations preceding rewards, and 92 ± 14% for the 48 reward responses (Fig. 3).

View larger version (9K): [in this window] [in a new window]   Fig. 3. Histograms of percent change in activity rate for neurons discriminating between different reward magnitudes. Neural data were obtained from trials in which animals were working for the highest level magnitude of reward. Each histogram shows a different activation type, with all 5 response types represented (Instruction, Delay, Trigger, Reward Expectation, and Reward).

Regression analysis between spiking rate and reward magnitude

To better understand the relationship between single unit spike rate and reward magnitude, a regression analysis was completed. Forty-eight neural responses were tested using all three liquid reward amounts. Seven of these neural responses showed significant activity differences for both reward combinations (High Reward vs. Medium Reward and Medium Reward vs. Low Reward), while 41 responses had a significant difference between one reward magnitude combination. Three types of relationships were seen between neural activity and reward magnitude. One population had the greatest activity to the high-level reward magnitude and showed a monotonic positive relationship between reward drop size and firing rate (n = 21; Fig. 4A; y = 0.67 + 2.375 × x; r² = 0.89; n = 21; t = 2.92; P < 0.001). Of these 21 neurons showing a descending magnitude relationship (High > Med > Low), 5 responded at the time of instruction picture presentation, 4 neurons responded during the delay period, 2 neurons responded during trigger presentation, 6 neurons responded prior to the reward delivery and after the movement execution, and 4 responded during reward delivery. Of these 21 responses, 4 responses had a significant difference between both reward magnitude combinations while 17 of the responses had a significant difference between one combination and an activity difference trend in the similar direction.

View larger version (14K): [in this window] [in a new window]   Fig. 4. Correlational analysis between the single neuron firing rate and magnitude of reward. The analysis used neurons that had only 1 significant difference between the 2 different magnitude trial types, with a trend in the same direction for the 2nd magnitude pair (n = 30) plus neurons that had 2 significant differences in activity between the different reward magnitude trials (n = 9). Neurons were separated into 2 groups: 1 with the greatest activity in the spike rate in the largest reward magnitude trials (A: n = 21; High > Medium > Low) and the other with the greatest activity in the low reward magnitude trials (B: n = 18; Low > Medium > High). The linear regression result was significant for both groups, with a high correlation coefficient indicating that spike rate is linearly related to magnitude in a rigorous manner. Neural activations used were taken from all the different response types.

The second population of cells analyzed using regression analysis showed the greatest activity in trials for the low level reward drop size and the least activity in trials for the large reward drop size (n = 18). These 18 neurons had either a significant difference in both reward comparison trials (n = 3) or a significant difference in one reward comparison with a trend in the similar direction in the second magnitude comparison (n = 15). Regression analysis showed that these activations had a clear negative monotonic relationship to reward magnitude (Fig. 4B; y = 2.25  1.787 × x; r² = 0.902; n = 18; t = 5.57; P < 0.0005). Of these 18 neurons used in the ascending reward magnitude comparison (Low > Medium > High), 2 were instruction responses, 6 were responses during the extended delay period, 4 were responses during the trigger presentation, 3 were activations prior to the reward delivery, and 3 were responses during reward acquisition.

A final category of cells (n = 9) that were tested using all three reward magnitudes showed a nonlinear relationship between reward magnitude and neural activity. These neural responses had an activity change in one reward magnitude level compared with a similar response to the other two reward magnitude levels. This quadratic relationship was characterized by the highest neural response at the medium level reward magnitude in eight cases and with the highest activity at the largest reward amount in a single case. Four of these responses were to the instruction cue, one was prior to the trigger stimulus, two were prior to the reward outcome, and two were during the reward delivery.

Responses to instructions

A total of 84 neurons responded to the instruction cues with transient activations that subsided within 500 ms after stimulus disappearance (Table 2). Of these, 61 neurons (73%) discriminated between the trials for different reward liquid amounts. Spatial discrimination between the left and right instruction picture was seen in only three neurons, with all three cells showing differential activity between trials for differing reward magnitudes, all of them on a single side of arm movement. The remaining 58 cells discriminating between reward magnitudes were spatially nonselective. Eleven of these neurons were tested with all three levels of reward magnitude and showed consistently higher activity for a particular magnitude level (Fig. 5). A subset of seven nonspatial neurons was tested in nonmovement trials, and five of them showed greater activity in movement than nonmovement trials (Fig. 6). We tested 24 neurons in trials with different picture sets and found that responses in 18 of them maintained the same relationship to the picture-magnitude association and varied insignificantly between different pictures predicting the same reward magnitude (Fig. 7).

View larger version (15K): [in this window] [in a new window]   Fig. 5. Differential responses to reward magnitude predicting instruction images in a nonspatial caudate neuron. This neuron failed to discriminate between different spatial locations, but discriminated between high, medium, and low reward magnitudes showing the greatest activity for the image paired with the largest magnitude reward. Pictures above the histograms show the instruction image that was used for the different reward amounts. Trials alternated semi-randomly between 2 different reward magnitudes and 2 instruction positions and are separated for analysis while pooling over the left and right instruction positions. There were no significant differences between these left and right spatial positions. Trials are rank ordered according to instruction-reward intervals. Vertical scales show impulses per bin. These histograms, as well as the following neural activity histograms (Figs. 5-13), are presented using a binwidth of 25 ms.

View larger version (24K): [in this window] [in a new window]   Fig. 6. Nonspatial movement vs. nonmovement instruction-related activity in a caudate neuron. A: activity to the instruction image discriminates between trials for low reward amount vs. high reward amount. The instruction image used in the trials is shown above the histogram. The neuron failed to discriminate between left and right movement trials. B: activity difference for this neuron disappears during the set of nonmovement control trials. In this trial type, the animal must withhold the movement to receive the liquid reward. These trials are separated for analysis. Trials for the 2 reward amounts, 2 spatial positions, and movement or nonmovement are presented in a semi-random fashion to the animal. Trials are rank-ordered according to instruction-reward intervals.

View larger version (60K): [in this window] [in a new window]   Fig. 7. Nonspatial discriminating activity time-locked to instruction presentation found in the ventral striatum. This neuron shows differential activity to the instruction picture depending on the associated reward magnitude. A: activations are highest to the instruction that signals medium-size reward compared with the activations related to the instructions for low or high reward. B and C: a 2nd and 3rd picture set was used to determine whether activity differences were due to the differences in the instruction pictures or due to the associated outcome. In both trial sets, the activity remained linked to the instruction cue that signaled medium-sized reward, indicating that the activation difference was related more to the predicted outcome to be received then differences in physical properties of the images viewed.

Activations during the instruction-trigger delay

A total of 91 task-related neurons showed activations that started during the instruction-trigger period and terminated >500 ms after instruction cue departure, either before or immediately after the trigger stimulus (Table 2). Activations in 56 of these 91 neurons (62%) differed between reward magnitudes. Activations in 4 of these 56 responses differed between the two spatial locations (left vs. right) of the cue (Fig. 8). Within this spatially selective group, activations either failed to discriminate between the different reward magnitudes on the alternate spatial location (2 neurons) or were entirely absent on the alternate spatial location (2 neurons). The remaining sustained activations were spatially nonselective (Fig. 9; n = 52). A subset of five of these activations was tested in nonmovement trials. Of these, three cells discriminated between reward magnitudes in movement trials as well movement trials.

View larger version (21K): [in this window] [in a new window]   Fig. 8. Spatial delay activity discriminating between medium and high reward amounts in a ventral striatum neuron. Activity during the instruction-trigger delay discriminated between trials for left (A) and right (B) movement targets showing preferential activity to right target trials when the animal is working for the medium-size reward and not the larger (High) reward. Insets show the instruction pictures that were used on left or right trial types. All 4 trial types were interspersed in a semi-random fashion.

View larger version (27K): [in this window] [in a new window]   Fig. 9. Nonspatial delay activity of 2 different ventral striatal neurons discriminating between 3 different reward magnitudes. This neuron failed to discriminate between left and right movement targets. A: neuron shows significantly greater activity during the delay period during trials for High Reward compared with the trials for lower reward amounts. B: 2nd ventral striatal neuron shows the opposite relationship between firing rate and reward magnitude. Here we show an example of a neuron that has the greatest activity during trials for the Low Reward vs. trials for the larger reward amounts.

Activations following the trigger stimulus

A total of 112 task-related neurons showed activations that closely followed the appearance of the movement trigger stimulus (Table 2). Activations in 31 of these neurons discriminated between rewards (28%). Ranking of trials according to the interval between the trigger stimulus and movement onset allowed us to determine the temporal relationships between these two events. Accordingly, the activations were classified as movement-related (73 neurons, 19 of them reward-discriminating; Fig. 10), trigger responses (9 neurons, 2 of them reward magnitude discriminating), or undefined (30 neurons, 10 of them reward magnitude discriminating). Four of the movement-related and reward magnitude discriminating responses (n = 19) differentiated between left and right movement targets, all of them on one side only. Activations with movement targets on the other side either failed to discriminate (2/4 neurons) or were entirely absent (2/4 neurons). None of the other responses showed spatial selectivity (n = 39). Out of these activations, 16 were tested in nonmovement trials. Fourteen showed higher responses in movement compared with nonmovement trials. Of these, four neurons discriminated between reward magnitudes in movement trials (Fig. 11).

View larger version (15K): [in this window] [in a new window]   Fig. 10. Reward magnitude discrimination in a movement-related putamen neuron. Activity did not differ between left and right movement targets but was time-locked to the key off more than the trigger onset indicating that the activity could be related to the movement in a nonspatial manner. The activity was highest in trials for the smallest reward (Low Reward) and decreased substantially in the other trial types using the same motor components. Trials alternated semi-randomly between the left and right movement targets and between the 3 reward magnitude types. Trials are rank-ordered according to reaction time.

View larger version (23K): [in this window] [in a new window]   Fig. 11. Reward-discriminating activations after the movement trigger onset in a caudate neuron. This neuron showed greater activity for the smaller reward quantity (Low Reward) compared with the larger reward drop size (Medium Reward), and this activity was observed solely in the movement trial types (A) and not in the nonmovement trial types (B). Six different trial types were presented in a semi-random alternation between left and right target, between Low and Medium reward types, and between movement and nonmovement trials. Trials are rank-ordered according to the time interval between trigger onset and key offset.

Activations preceding rewards

A total of 109 task-related neurons showed activations that began prior to reward delivery (Table 2). These activations remained present until the reward was delivered and terminated <500 ms afterward. These responses occurred in both movement and nonmovement trials. Prereward activations in 48 of the 109 neurons (44%) discriminated between rewards. In 28 of 48 (58%), the greater activity was to the larger reward magnitude (Fig. 12), and in 20 of 48 (42%) neurons, the greater activity was seen to the smaller reward magnitude. Previous work investigating licking actions has supported the idea that many of the reward-discriminating, anticipatory neuronal responses were not due to the differences in anticipatory licking (Hassani et al. 2001; Hollerman et al. 1998). This observation was presently corroborated by the subset of neurons that showed greater activity in trials with smaller reward, because smaller rewards were usually accompanied by fewer anticipatory licking.

View larger version (20K): [in this window] [in a new window]   Fig. 12. Reward discriminating activation preceding the different juice magnitudes in a single caudate neuron. This neuron discriminated between High Reward and Medium Reward, between Medium Reward and Small Reward, and between High Reward and Small Reward. It failed to differentiate between left and right movement targets. All trial types alternated semi-randomly and are separated for analysis. Trials are rank-ordered according to the intervals between instruction onset and reward delivery.

Activations following rewards

A total of 104 task-related neurons showed responses that followed the delivery of the reward drop to the mouth and subsided prior to the instruction presentation of the subsequent trial (Table 2). Activations showing close temporal relationships to licking movements were discarded from the sample. These responses were characterized by phasic activity during licking, time-locked to tongue extensions and retractions and appeared similar to previously reported tongue-movement related activity within the striatum (Apicella et al. 1991). The reward responses varied insignificantly in activity level between left and right lever target trials and occurred in both movement and nonmovement trials. Activations in 48 of the 104 (46%) neurons discriminated between reward magnitudes (Fig. 13), irrespective of the side of the movement target. These 48 reward magnitude discriminating neuronal activations could occur in trials in which there were no major differences in licking behavior. All of the activations tested (n = 12) were also observed in nonmovement trials.

View larger version (25K): [in this window] [in a new window]   Fig. 13. Differential activity of a ventral striatal neuron following reward delivery. This neuron shows higher activity to the reception of the smallest reward out of the 3 reward outcomes. The 3 reward magnitude trials were presented in a semi-random fashion and are presented separately for analysis. The trials are rank-ordered according to the time between instruction onset and reward delivery.

Recording positions

Histological reconstructions of recording positions revealed that neurons were sampled in caudate nucleus, putamen, and ventral striatum, including nucleus accumbens between rostrocaudal levels A18 and A25 and mostly rostral to the anterior commisure (Fig. 14). Recordings were made throughout the entire dorsoventral extent of these structures and were mediolaterally concentrated around the internal capsule.

View larger version (29K): [in this window] [in a new window]   Fig. 14. Positions of discriminating neurons with different task relationships in the 2 monkeys. The 3 portions of the striatum are divided by the dashed lines (Cd, caudate nucleus; Put, putamen; V St, ventral striatum including nucleus accumbens; AC, anterior commissure). The 5 different symbols each refer to a particular task-related response as defined in the text (see METHODS). A 6th group, labeled Mixed, includes those neurons that showed activations to more that 1 event within the delayed response trial.

Reward magnitude-discriminating neurons were found in the caudate (41 of 83 task-related neurons, 49%), putamen (99 of 175 neurons, 56%), and ventral striatum (64 of 116 neurons, 55%). Activations preceding the trigger stimulus were more prevalent in the putamen compared with the caudate nucleus or ventral striatum for both the reward discriminating neurons (Fig. 15; P = 0.006; ² test) and the nondiscriminating neurons (Fig. 15; P = 0.0001, ² Test). Their distribution failed to vary significantly among the rostrocaudal levels explored (Fig. 16; A18-A25; P = 0.19). The distribution of spatially discriminating cells varied significantly between these structures (P = 0.0005), being lower in the ventral regions of the striatum (0 neurons with instruction, delay, or trigger activations) compared with dorsal striatum (caudate = 2%, putamen = 5%). The distribution of reward magnitude-discriminating neurons among the spatially discriminating neurons varied insignificantly among the three structures (P = 0.18). Very similar results were obtained in previous studies examining the neuronal discrimination between different rewards (Hassani et al. 2001) or between reward and no reward trials (Hollerman et al. 1998).

View larger version (21K): [in this window] [in a new window]   Fig. 15. Regional distributions of neurons in 3 different striatal subregions for discriminating neuron pool (A) and nondiscriminating neuron pool (B). The anterior striatum was divided into caudate nucleus, putamen, and ventral striatum, and the total number of neurons found in each region was tallied and compared within the 5 different task-related response types. There were significantly higher numbers of neurons in the putamen in 3 different responses, including responses following the trigger stimulus in the pool of discriminating neurons and responses following the trigger and preceding the reward in the nondiscriminating pool of neurons (P = 0.006, P = 0.0001, and P = 0.003; 2 test). Black filled bars denote activities prior to the event (trigger or reward) and gray-filled bars denote number of neurons following the event. PUT, putamen; CD, caudate nucleus; VST, ventral striatum including nucleus accumbens. y axis scales vary between the histograms for the 2 pools of neurons.

View larger version (39K): [in this window] [in a new window]   Fig. 16. Rostrocaudal distribution of task-related activations in the 2 monkeys for the pool of discriminating neurons (A) and nondiscriminating neurons (B). No significant differences were found for the different task-related responses for the rostrocaudal distribution. The analysis of the rostrocaudal distribution divided the striatum into 4 AP levels (A18-A19, A20-A21, A22-A23, and A24-A25). The majority of the neurons were found in the middle regions of the anterior striatum (A20-A21 and A22-A23) for all response types. y axis scales vary between the Instruction/Trigger and the Reward histograms.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

The present results show that single neurons within the anterior striatum distinguish between minute differences in reward magnitude (<0.1 ml). These discriminatory activations were time-locked to the predictive stimuli, during the delay periods and preceding and following the liquid reward delivery. In addition, they were observed preceding and following the arm movement to obtain the reward. The percentage of neurons that showed a task-related activation in the present study (374/2,000; 19%) is comparable to previous findings in studies examining striatal activity during the delayed response task (Hassani et al. 2001; Hollerman et al. 1998). The percentage of these task-related activations with an activity difference between rewarding magnitudes was 55%. This number is exceedingly higher than the percentage of striatal neurons that were found to discriminate between two different rewarding outcomes (34%; Hassani et al. 2001) and substantially lower than the proportion of neurons that discriminated between trials for reward versus no reward (97%; Hollerman et al. 1998). We believe that these levels of responsiveness represent meaningful signals that potentially have a large influence on behavioral reactions. Recent neuroanatomical data on the convergence and connectivity of corticostriatal inputs supports the idea that small samples of striatal neurons can carry a heavy information processing load. It has been shown that individual neurons in the striatum receive a great amount of information in terms of the number and variety of inputs from cortical and subcortical inputs. Individual cortical axons are unlikely to synapse onto striatal neurons within the same subregion but more likely to synapse onto neurons spatially segregated (Kincaid et al. 1998). This type of neuroanatomical organization could enable low numbers of neurons to be sufficient for information processing as the level of integration of information in terms of the number of features being encoded becomes larger or the degree of difficulty involved in making stimulus/outcome discriminations becomes greater.

It is still unknown exactly how these differences in striatal neural activity influence behavioral reactions of the animals as they work for different reward outcomes. The neural activations at different timepoints discriminating between outcomes could influence behavior in several different manners. For example, neural activity surrounding the predictive cue could be more important in mediating cue-reward associations and neural activity surrounding the movement could be key for the coding of reward-dependent motor reactions. One well-known finding in behavioral work is that animals performing tasks for different reward outcomes display dissimilar general behavioral strategies in acquiring the disparate outcomes.

Behavioral discrimination

Psychologists and other behavioral scientists have long noted that animals alter their anticipatory behavioral strategies dependent on the expected reward outcome to be received (Crespi 1942; Flaherty 1996; Tinklepaugh 1911; Tolman 1932). This behavioral discrimination has been proposed to aid animals in the cost-benefit analyses that occur during normal foraging (Kirshenbaum et al. 2000). Tolman (1932) argued that animals learn expectancies that occur prior to the outcome that guide the actions. When outcomes are acquired, the end result either confirms or disconfirms those expectancies. Future behavior is then guided by the degree of the discrepancy between expectation and outcome. In this role, expectancies are much more efficient if they contain information about the anticipated outcome along with the memories of the past discrepancies following particular behaviors. Between sets of trials with different outcomes, relative reward effects can occur. Important properties of the tasks that influence the behavioral expressions between these trials include 1) the disparity between the outcomes (as rewards become more disparate, the behavioral reactions become discernibly different prior to reward reception) (Bloomfield 1967; McSweeney 1975); 2) the similarities between the predictive cues (as they become more or less similar, anticipatory expressions are altered) (Blough 1988; Mackintosh 1974); and 3) the time delays between events in the task (as the delays change between events and rewards, the anticipatory actions are altered) (Williams 1983). In the present study, temporal delays were similar between task events for all trials and the predictive stimuli were shown to be highly discriminative from one picture set to the next for signaling the different outcomes. Disparity between outcomes remains as the potential main determinant of the behavioral discrimination seen in the present study and reinforces the idea that the observed neural discrimination potentially modulating general behavioral differences are rich in details concerning the upcoming outcome.

Neural discrimination

We used a number of control tasks to better understand the relationship between neural activity and the reward outcome difference. These tasks were meant to help distinguish between possible influences of sensory or motor aspects of the task and the difference in magnitude.

MOTOR RESPONSES AND NEURAL DISCRIMINATION. It could be that the discriminatory activity in the striatum reflects differences in the movements made by the monkeys in different trials. Striatal activity has been shown to be related to forearm and eye movements (Delong et al. 1986; Hikosaka et al. 1989a). Despite the reliable movement differences between trials for smaller and larger rewards, the observed changes in striatal neurons were not primarily due to the particular motor components. We ranked trials according to reaction time speed and found no consequent ranking for the activity rate of the individual neurons (Figs. 10 and 11). In many cases, we found that the reaction times were slower and lick rate less, but the neural response was significantly increased. Similar findings were observed when monkeys were working for two different types of juice rewards (Hassani et al. 2001). In terms of the present work, we hypothesize that the expectation generated at the level of the striatum is not directly linked with any one motor parameter but influences a host of diverse behavioral reactions that include eye, arm, and whole body movements.

SENSORY INPUT AND NEURAL DISCRIMINATION. We used a set of different instruction pictures associated with each reward amount to assess the contribution of image properties to the reward-discriminating activations. Previous work has shown that there is visual or object-related activity in the tail of the caudate nucleus (Brown et al. 1995). Neural activations related to visual cues have been found in ventrolateral prefrontal cortex and inferotemporal cortex (Liu and Richmond 2000; Miller et al. 1996). These areas of cortex send projections to regions of the striatum (Selemon and Goldman-Rakic 1985) and could be important in the production of the reward influence associated with the visual cues seen in the present study. Approximately 75% of our neurons tested with multiple picture sets for the same reward magnitude maintained a similar profile of responsiveness between trial blocks, i.e., they continued to respond to the cue that predicted the same outcome (large or small reward) independent of the changing image (Fig. 7). This supports the idea that within a subset of striatal cells, the discrimination occurred on the basis of the upcoming reward magnitude rather than the sensory features of the different images presented. In the orbitofrontal cortex, the influence of the reward seems greater due to the fact that an even higher percentage of neurons maintained responding across different image presentations (Tremblay and Schultz 1999). The anterior striatal regions recorded from in the present study receive a dense projection from the orbitofrontal cortex that could be very important in the production of the observed responses (Haber et al. 1995).

AROUSAL AND NEURAL DISCRIMINATION. It has been hypothesized that general arousal levels shift when expectations for different rewards occur (Black 1968). These arousal differences may contribute to the changes in neural activity by individual striatal neurons. This idea is clearly supported whenever there is a positive relationship between firing rate and reward preference in terms of liquid type or amount, but the present results show many cases where there is an inverse relationship between reward magnitude and neuronal firing rate (Figs. 8, 10, 11, and 13). In the inverse cases, the greatest activity was observed in trials when the animal expected the smaller magnitude reward. This indicates that arousal effects on neural activity based on a simple linear relationship between reward magnitude and arousal could not explain our results. Similar to our results, previous work in striatum has found greater neural activity at either end of an established preference hierarchy, supporting the idea that changes in arousal are not primarily responsible for the differences in reward processing in this forebrain structure (Hassani et al. 2001; Hollerman et al. 1998). Others have found that activity of single striatal neurons is consistently related to the predictability of the stimulus or the reward and not dependent on the arousal of the animal (Bowman et al. 1996; Liu and Richmond 2000; Shidara et al. 1998). The possibility still exists that arousal is not simply a unimodal phenomenon related to either nonpreferred or preferred stimuli or outcomes in a linear and interdependent fashion, but could possibly be related to both types of outcomes in an orthogonal manner, which might lead to a bimodal distribution of several arousal peaks at different points along the preference hierarchy. If something like this relationship exists and interacts with striatal processes, then there is still a possibility for arousal states to modify the neural activity observed in the present study.

Striatal afferents and reward expectancies

Where does the information concerning the reward and movement arrive from that influences these striatal activations? Movement-related inputs arrive from a host of motor areas in the frontal cortex, including the supplementary motor area, premotor cortex, and primary motor cortex (Alexander et al. 1986). In addition, a large input arrives from the anterior cingulate cortex in the medial wall of the prefrontal area (Kunishio and Haber 1994). These frontal motor areas have been found to integrate information concerning environmental events with motor processing (Boussaoud et al. 1995). In the premotor cortex, neurons have been shown to code for movement intention separate from sensory attention (Boussaoud 2001). In the medial wall, cortical areas have been shown that integrate motor sequence information with particular outcomes; neurons in the anterior cingulate cortex respond during particular motor sequences used to acquire specific reward outcomes (Shima and Tanji 1998). Recently, neural signals in this region have been shown to be related to the strength of the reward expectancy as progression toward the outcome was predictably changed between different schedules of reward responding (Shidara and Richmond 2002). These results indicate that at the cortical level integration occurs between motor information and other information. Whether this level of integration occurs prior to or as a consequence of striatal processing is still an open matter. Cross-correlograms of simultaneous medial cortical and striatal activity during an operant task revealed ventral striatal neurons with an activity preceding cortical activity (Chang et al. 2000). This suggests that the striatum may be the source of certain cortical integrative activity within a well-learned behavioral paradigm.

Neurons in other frontal regions detect rewards and reward-predicting stimuli. In particular, the orbitofrontal cortex has been shown to contain neurons that differentiate well between different rewarding items, possibly on the basis of relative reward value (Critchley and Rolls 1996; Gallagher et al. 1999; Hikosaka and Watanabe 2000; Thorpe et al. 1983; Tremblay and Schultz 1999). Recent functional magnetic resonance imaging (fMRI) data show that distinct regions of human orbitofrontal cortex are activated by rewarding stimuli with the degree of activation related to the magnitude of the upcoming, expected reward (O'Doherty et al. 2001). The striatum could employ this information in the production of appropriate motor responses based on how the orbitofrontal represented an outcome relative to other outcomes in the immediate environment.

The dorsolateral prefrontal cortex has also been shown to be involved in reward processing and subsets of neurons in this region show sustained activation between rewards and their predictive stimuli (Fuster 2000). These activations can be dependent on the reward type (Watanabe 1996) or magnitude (Leon and Shadlen 1999). Spatial information is known to be a critical component of dorsolateral prefrontal function in spatial delayed response tasks (Funahashi et al. 1989), and it could be that this region merges spatial and reward information together to be sent to subcortical structures for further processing. Other researchers have found this type of integration in the medial prefrontal cortex (Pratt and Mizomori 2001). Lesions to this same region impair adaptive behavioral changes following reward magnitude shifts in a spatial task (DeCoteau et al. 1997).

The amygdala sends direct input to the striatum (Russchen et al. 1985) and has been shown to be an important structure in reward processing (Everitt et al. 1999). In the primate, it has been shown that single amygdala neurons respond to rewards and the predictive signals related to those rewards (Nishijo et al. 1995; Rolls 1992). It is thought to be involved in the associative learning about reward-stimulus relationships in the environment (Parkinson et al. 2001; Shoenbaum et al. 2000). Inactivation of the amygdala decreases memories for reward magnitude shifts (Salinas and McGaugh 1993, 1996), and amygdala neural responses to rewards and reward-related stimuli are influenced by changes in the internal state (Cromwell et al. 2001; Rolls 1992). Amygdala neurons most likely process attributes of stimuli during learning, and once learned, the structure may mediate general influences involved in emotional processing. For the motivational system of feeding and food seeking, these influences include the cue-reward associations predicting food availability or the degree of food deprivation.

Role of striatum in reward expectation

What roles could the striatum be performing in the processing of rewards through neural expectancies? We hypothesize that the striatum is critical for linking precise reward information with a diverse set of movement outputs expressed together as a general behavioral reaction. The idea that the reward information can be very precise is critical to understand the ability of striatal processing in modulating goal-directed action. This essential aspect of striatal functioning most likely depends on the multi-leveled regions of convergence within the corticostriatal system (Flaherty and Graybiel 1991, 1993; Haber and McFarland 2001; Haber et al. 2000; Kincaid et al. 1998) that allows these cells to access and use a variety of information concerning the goal objects to activate motor strategies that can be individually tuned to the particular behavioral context. This potential influence can be greater than mere selection because the output contains an invigoration of a particular motor response in the face of multi-faceted environmental choices (Cardinal et al. 2001; Everitt et al. 1999). A potential invigorating component may arise from information regarding internal state changes or from information about the relative value of an external event. It is this diversity of inputs that allows these neurons to process fine details of the stimulus or motor event. This same level of detail in processing that spans motor and sensory realms has not been found to such a degree in cortical or other subcortical structures.

A second critical aspect of striatum processing is the ability to link the general and specific inputs to a wide range of motor reactions. The striatum does not seem to be intimately involved in the production of discrete motor acts (Albertin et al. 2000; Cromwell and Berridge 1996; Lesuzuk and Flaherty 2000). Instead, it seems to be important in the integrating diverse sets of information with general motor plans embedded inside complex movement sequences (Cromwell and Berridge 1996; Lesuzuk and Flaherty 2000). This function is enabled through the broad, parallel streams of outputs from the striatum onto several frontal cortical regions involved in motor processing (Alexander et al. 1986).

These features could allow the striatum to utilize information in a distinct manner compared to cortical subsystems. The putative nature of this distinctiveness can be highlighted by comparing possible differences in functional significance but not expectancies and working memories. Sustained activations in the prefrontal cortex and parietal cortex have been labeled as working memories involved in the decision making process (Kim and Shadlen 1999; Platt and Glimcher 1999; Wallis et al. 2001); therefore they should be critical for the production of the appropriate motor response within that trial setting. Recent data have shown that sustained activations within the dorsolateral prefrontal cortex can code precise information of parametric value when the parametric information must be memorized to perform the task correctly (Braver et al. 1997; Romo et al. 1999). In contrast, the influence of an expectancy does not seem to necessarily include a role in coding the instrumental properties of the task. The reward information utilized by striatal neurons in this study, as well as other studies (Hassani et al. 2001; Hollerman et al. 1998), is not crucial information needed for proficient task performance. In this way, the expectancy functions not as a key ingredient in decision-making but as a comparator and amplifier that can invigorate behavioral reactions to particular objects in the environment. Neural activations found in anterior cingulate cortex have been proposed recently to function as expectancies for particular rewards (Shidara and Richmond 2002), and these activations are related to the actual progression to the outcome through a predictable series of trials. These predictive cues, similar to the ones in the present study, were not instrumental for accurate behavioral output. Instead, they represented noninstrumental aspects, including delay to reward reception and the response effort needed to obtain the reward. In the present study, the cues and the elicited expectancies seem to represent the amount of absolute reward and possibly the relative magnitude. The issue of relativity has been a critical aspect in the study of sensory and perceptual systems. It is now time to transfer these ideas more to the study of the physiology of reward and other motivational systems. Future studies examining the factors comprising expectations and how these neural activations influence behavior will be vital to better defining the "limbic processing" involved in the motivational-to-motor-interface function of the striatum.