We recorded the activity of single neurons in the posterior parietal cortex (area LIP) of two rhesus monkeys while they discriminated the direction of motion in random-dot visual stimuli. The visual task was similar to a motion discrimination task that has been used in previous investigations of motion-sensitive regions of the extrastriate cortex. The monkeys were trained to decide whether the direction of motion was toward one of two choice targets that appeared on either side of the random-dot stimulus. At the end of the trial, the monkeys reported their direction judgment by making an eye movement to the appropriate target. We studied neurons in LIP that exhibited spatially selective persistent activity during delayed saccadic eye movement tasks. These neurons are thought to carry high-level signals appropriate for identifying salient visual targets and for guiding saccadic eye movements. We arranged the motion discrimination task so that one of the choice targets was in the LIP neuron's response field (RF) while the other target was positioned well away from the RF. During motion viewing, neurons in LIP altered their firing rate in a manner that predicted the saccadic eye movement that the monkey would make at the end of the trial. The activity thus predicted the monkey's judgment of motion direction. This predictive activity began early in the motion-viewing period and became increasingly reliable as the monkey viewed the random-dot motion. The neural activity predicted the monkey's direction judgment on both easy and difficult trials (strong and weak motion), whether or not the judgment was correct. In addition, the timing and magnitude of the response was affected by the strength of the motion signal in the stimulus. When the direction of motion was toward the RF, stronger motion led to larger neural responses earlier in the motion-viewing period. When motion was away from the RF, stronger motion led to greater suppression of ongoing activity. Thus the activity of single neurons in area LIP reflects both the direction of an impending gaze shift and the quality of the sensory information that instructs such a response. The time course of the neural response suggests that LIP accumulates sensory signals relevant to the selection of a target for an eye movement.
Primates use vision to guide their interactions with the environment. In wakefulness, the brain generates a steady stream of decisions to shift the gaze, to position the body, and to grasp, avoid, or classify objects, often with the guidance of data from the visual cortex. Unless an action is purely reflexive or purely capricious, a higher level of information processing must link sensation to action. Sensory data must beinterpreted to execute, revise, or delay pending action. The goal of this study is to investigate the neural underpinnings of one such interpretive mechanism: a simple decision process in a two-alternative, forced-choice psychophysical paradigm.
We trained monkeys to discriminate opposed directions of motion in a stochastic random dot display and to report the perceived direction with a saccadic eye movement to one of two visual targets. At least three processing stages must be engaged during each trial the monkey performs (Fig. 1). First, a sensory process must extract motion information from the visual image and represent the outcome within the visual cortex. For our task, the relevant representation of motion resides largely in areas MT and MST of extrastriate cortex (Britten et al. 1992,1996; Celebrini and Newsome 1995;Croner and Albright 1999; Newsome and Paré1988; Salzman et al. 1992; Shadlen et al. 1996). Neurons in MT and MST generate smoothly varying responses that reflect the amount of motion energy within a specific band of velocities (direction and speed) to which they are tuned (Albright 1984; Maunsell and Van Essen 1983; Simoncelli and Heeger 1998; Zeki 1974). Second, the map of motion direction in MT and MST must be interpreted, or read out, to form a categorical decision: is the net motion flow in direction A or direction B? Third, after a decision is formed, it may need to be stored in working memory until an operant response is programmed and executed. In our task, neural signals for guiding the operant response must ultimately emerge from eye movement–related structures such as the superior colliculus, the frontal eye field, and the lateral intraparietal area (LIP) of the inferior parietal lobe, areas that have been studied extensively over the past few decades (for reviews, seeAndersen et al. 1992; Colby and Goldberg 1999; Schall 1995). We therefore have a reasonable base of knowledge concerning the sensory and motor processing stages that must be engaged during performance of the task, but we know virtually nothing concerning the key cognitive stage of decision formation (see also Romo and Salinas 2001).
As an initial step toward analysis of the decision process, we have studied the activity of neurons in LIP that carry high-level signals appropriate for identifying salient visual targets and ultimately for guiding saccadic eye movements. Many neurons in LIP modulate their level of activity when there is sufficient information to plan a saccade, even when execution of the saccade may be delayed by several seconds (Colby and Goldberg 1999; Mazzoni et al. 1996; Snyder et al. 2000). Our central question is whether the activity of these neurons can provide insight into the process of decision formation during performance of our psychophysical task. Differentiating decision-related activity from strictly sensory activity is reasonably easy. By requiring the monkey to discriminate weak, noisy motion signals near psychophysical threshold, we create a situation in which the decision varies from trial to trial for repeated presentations of the same motion stimulus (i.e., the monkey decides correctly on some trials and incorrectly on others). To a first approximation, sensory activity will reflect the motion in the stimulus irrespective of what the monkey decides, whereas activity in higher level circuits that interpret the motion signals should vary strongly with the monkey's decision.
Differentiating decision-related activity from strictly motor activity, however, is not so straightforward. In a trivial sense, all motor signals are decision-related in that they reflect the outcome of the decision process. The key problem is to differentiate processing stages in which the decision is actually formed and represented from stages that simply represent a movement to be executed. We have adopted two tactics to gain experimental leverage on this issue. First, we have introduced an instructed delay period between presentation of the motion stimulus and the “go” signal to execute the saccadic eye movement. This tactic delays overt motor activity until the end of the trial, thereby separating the period of motion viewing (hence the decision) from motor execution. Second, we have taken advantage of a fact that all psychophysical subjects know well: all decisions are not created equal. Subjects are certain of decisions made on the basis of strong sensory information but are quite doubtful of decisions made on the basis of ambiguous evidence. We assume that neural circuits intimately linked to the process of decision formation should reflect this level of certainty either in the amplitude or timing of decision-related activity (Basso and Wurtz 1998). In other words, decision-related activity should bear some signature of the intensity of the sensory stimulus.
We have found that some neurons in LIP are plausible candidates for participating in the decision process. These neurons generate sustained activity that predicts the impending saccade, and thus the monkey's decision. Both the amplitude and timing of this activity reflect the certainty of the decision and cannot be accounted for by any parameter of the eye movement itself that we have investigated.
Subjects, surgery, and daily routine
We performed experiments on two adult rhesus monkeys (Macaca mulatta, 1 male and 1 female) weighing 8–9 kg. The monkeys were surgically implanted with a head-holding device (Evarts 1968), a scleral search coil for monitoring eye movements (Judge et al. 1980), and a recording cylinder over the intraparietal sulcus. After recovery from surgery, the animals engaged in daily training or experimental sessions lasting 2–6 h. The monkeys were trained to perform a two-alternative, forced-choice direction discrimination task near psychophysical threshold. The monkeys were also trained on a variety of fixation and saccadic eye movement tasks as described below.
The monkeys worked for liquid rewards, and their daily water intake was therefore controlled. All surgical and behavioral procedures were in accordance with the U.S. Department of Health and Human Services (National Institutes of Health) Guide for the Care and Use of Laboratory Animals (1996).
Visual stimuli were generated on a PC/486 computer using a Pepper SGT+ graphics board (Number 9 Computer) attached to a Sony multiscan monitor (60 Hz noninterlaced) placed 57 cm away from the monkey. The system displayed fixation and saccade targets as well as the dynamic random-dot motion stimuli used for the direction discrimination experiments. The motion display was similar to stimuli used in previous investigations (e.g., see Britten et al. 1992).
Random dots were plotted within a circular aperture of 5–10° diam. Each dot was displayed for one video frame and then replotted 50 ms (3 video frames) later either at an appropriate spatial displacement for apparent motion (typically 3–7°/s velocity) or at a random location. The probability that a particular dot would be displaced in motion is termed the motion coherence, expressed throughout the paper as a percentage. For example, if the coherence is 50%, then a dot that appears in frame 1 has a 0.5 probability of coherent displacement in video frame 4 and an equal chance of being randomly replaced somewhere else in the viewing aperture. Dots that first appear in video frame 2 are not seen in frames 3 and 4 and are subsequently plotted with the appropriate displacement (or randomly) in video frame 5, and so on. The dots were white on a black background and plotted at a density of 16.7 dots per deg2 per s, as in previous studies.
For some experiments, we used the same sequence of random dots for all trials at each coherence-direction combination. The manipulation did not lead to any detectable difference in the LIP response, and we have therefore combined these experiments with those in which a fresh random-number seed was used on every trial.
We recorded neural activity using tungsten microelectrodes (impedance 0.8–1.2 MΩ at 1 kHz; FHC) inserted into the cortex through a 23-gauge stainless steel guide tube that punctured the dura mater. The tip of the guide tube was either in the superficial layers of area 7a or in the intraparietal sulcus, outside of the cortex. The guide tube was held in place by a plastic grid fitted inside the recording chamber (Crist Instruments). The grid enabled us to record from the same location along the bank of the intraparietal sulcus for several days.
Signals were amplified and viewed on an oscilloscope screen. Single units were isolated on the basis of voltage waveform using a voltage-time window discriminator (Bak Electronics). The time of each action potential was stored on computer disk to the nearest millisecond, along with the time of trial events that identified the time of fixation, stimulus onset, stimulus offset, and saccade. Records of eye position were stored to disk (250 samples/s) on a portion of the experiments. Data acquisition and experimental control were accomplished using a PC/486 running a real-time data acquisition system (Hays et al. 1982). The trial events, spikes, and eye position data were analyzed off-line using software tools developed in Matlab (The Mathworks).
The primary goal of the study was to examine the responses of neurons during performance of a motion discrimination task similar to one used in previous investigations of areas MT and MST (Britten et al. 1992, 1996; Celebrini and Newsome 1994; Newsome and Paré 1988;Salzman et al. 1992). For the present study, neurons were selected on the basis of their responses during saccadic eye movement tasks, described below. For all tasks, the monkey was required to fixate a small red spot (the fixation point, FP) until its extinction. If at any time, the gaze fell outside of a 2 × 2° window centered on the FP, the trial was aborted. The window accommodated the small variation in eye position from trial to trial, but the monkey's gaze on any one trial was typically stable. A brief description of each task follows.
DELAYED SACCADES WITHOUT AND WITH MEMORY.
Neurons were screened by their responses in a delayed saccade task. On fixating a central spot, a bright red saccade target appeared in the periphery. The monkey was required to maintain fixation until the fixation spot was extinguished and was then required to make a saccade to the target within 500 ms. The delay period between onset of the saccade target and offset of the fixation point (“go” signal) was randomized from 0.5 to 2.0 s. We sampled LIP using this task, searching for those neurons that discharged during the delay period.
On isolating an appropriate cell, we identified the region of the visual field that led to robust responses during the delay period. We will refer to this region as the response field (RF) of the neuron. Some investigators would use the term receptive field or motor field, depending on whether the emphasis is placed on the response to visual targets or the preparation to make an eye movement (Mazzoni et al. 1996). In our experience these regions of the visual field were sufficiently congruent to warrant the more generic terminology (Barash et al. 1991b; Colby et al. 1996; Gnadt and Andersen 1988;Platt and Glimcher 1997, 1998).
To ensure that the delay-period activity we recorded was not due to the presence of the visual target, we also required the monkey to perform delayed saccades to remembered locations. This task is identical to the delayed saccade task, except that the target was turned off after 200 ms. The monkey was required to maintain fixation during the delay period (0.5–2.0 s) that ended with extinction of the fixation spot. The monkey was then allowed up to 500 ms to initiate a saccade to the remembered location of the target and was rewarded if the saccade endpoint fell within 4–8° of the cued location.
We explored the boundaries of the RF by changing the location of the saccade target. We did not attempt to map the extent of this region quantitatively, but we did identify regions of the visual field that failed to evoke delay-period activity during this task. We exploited this knowledge to place a second target, as well as the random-dot motion stimulus, outside the RF defined in this manner. This was easy to achieve in most instances because the RFs were eccentric and reasonably well circumscribed (median eccentricity was 9.6°; 87% were at least 5° from the fovea).
MOTION DISCRIMINATION TASK.
After delineating the boundaries of the RF, we set up a direction discrimination task after the design illustrated in Fig.2. One target, henceforth calledT1, was placed in the RF of the neuron under study, while a second target, T2, was placed well outside the RF (often in the opposite hemifield). The stimulus aperture was positioned so that the coherent dots moved toward one or the other target on each trial. We positioned the stimulus aperture so as to minimize stimulation of any visual receptive field.
The monkey performed a one-interval, two-alternative, forced-choice direction discrimination task. On each trial, motion was either toward or away from the RF, and the strength of the motion (% coherence) was also randomly varied to span psychophysical threshold. The random-dot motion was presented for 0.5, 1, or 2 s, followed by a delay period (duration 0.2–2.0 s) in which the monkey maintained fixation until extinction of the fixation point. The monkey then made a saccade to one of the two targets. If the coherent motion was toward the RF, the monkey was rewarded for an eye movement toT1; if the motion was away from the RF, the monkey was rewarded for an eye movement to T2. Importantly, the locations of the two saccade targets, the location of the stimulus aperture, and the axis of the motion discrimination were adjusted in each experiment according to the location of the neuron's RF.
In using this geometry, we created a situation in which a decision in favor of one direction of motion should be reflected by an increase in firing rate of the neuron under study because its RF would become the target of the subsequent saccade. Conversely, a decision favoring the other direction of motion, resulting in a saccade to the target outside the RF, should result in a decrease or exert no influence on the neuron's firing rate. The monkey's choices were tabulated as a function of motion strength to establish a psychometric function. Psychometric functions were fit with a cumulative Weibull function (Quick 1974) that estimates the probability of a correct choice as a function of motion coherence (COH) Equation 1Values for the two free parameters, α and β, were obtained using a maximum likelihood fitting procedure. We refer to the fitted value, α, as the discrimination threshold. At threshold (COH = α), the monkey is expected to make 82% correct choices. Across our experiments, the mean ± SE threshold was 15 ± 0.8% coherence (median 13.1%), The slope of the psychometric function was slightly greater than one (mean β = 1.1 ± 0.04, median 1.0), consistent with previous work (Britten et al. 1992). We are thus assured that the monkey used the weak motion cues in our stimuli to guide its selection of eye movements. For each neuron, we obtained data using the discrimination and delayed saccade tasks. When possible we also performed one or more of the control tasks described below.
PASSIVE VIEWING OF RANDOM-DOT MOTION.
We examined the response to random-dot motion during trials in which the monkey simply fixated. No saccade targets appeared on these trials, and the monkey was rewarded simply for maintaining fixation throughout the motion-viewing period. The dots appeared in the same location as in the discrimination task, outside the neuron's RF. The strength of motion was 51.2% coherence, which matched the strongest motion used in the discrimination experiments. This fixation task was often performed in a separate block of trials but was sometimes randomly interleaved with discrimination and delayed saccade trials. The task is the only one in which saccade targets do not appear shortly after fixation.
DELAYED SACCADES IN THE PRESENCE OF MOTION DISTRACTOR.
This task examines the response to visual motion during preparation of a saccadic eye movement that is specified by a single target. The task resembles the discrimination task with the important exception that only one saccade target appears throughout the trial. The motion coherence was 51.2% and the direction was toward or away from the target. Both the direction of motion and the target location were randomized and independent. The monkey was rewarded for making a saccade to the one target. Importantly, the direction of motion had no bearing on the monkey's reward. This task was always performed in a separate block of trials to distinguish it from the discrimination task. Because this task potentially reinforces a dissociation between motion direction and eye movement response, we included it only after obtaining data on the other tasks.
Raw data were stored as spike events timed to the nearest millisecond. These responses were collated into trials along with various time markers to compute standard peristimulus time histograms and rasters, and to count spikes occurring between trial events. Analysis was performed off-line using custom software developed in Matlab (The Mathworks). Each of the intervals comprising our trials (from target onset to motion onset, from motion onset to offset, and from motion offset to the extinction of the fixation point) contained a variable amount of time. We therefore present our data with respect to different event markers (e.g., motion onset). To compute summary statistics, we used the average spike rate between two trial events or in epochs aligned to common trial events (e.g., 1st 500 ms of motion-viewing period).
We performed standard comparisons of means using t- andF-tests. When examining results across the population of neurons in our data set, we applied multiple regression models in which cell identity was incorporated as an independent categorical variable. For example, to analyze the effect of motion strength on neural response (Fig. 9, A–D) we fit the model Equation 2where y is the spike rate measured in a designated epoch on correct T1 choice (or correct T2choices); COH represents motion strength;1 Iunit serves as an indicator function (Iunit = 1 if unit = i and 0 otherwise; such variables are often referred to as a dummy variables); and ε represents the residual error, which is assumed to obey a normal distribution. The fitted coefficient, β, along with its confidence interval provides an estimate of the effect of motion strength on response across the 104 neurons, allowing for differences in level of activity among the neurons (as estimated by the fitted coefficients α1 … α104).
To test whether saccade direction affects the response, we calculated the probability of obtaining an F-statistic under the null hypothesis, H 0: β = 0. TheF-statistic is derived from the extra sum of squares obtained by fitting a reduced model in which β = 0 (Draper and Smith 1966). If there are m data points and n neurons, then for two models that differ byk = 1 degrees of freedom Equation 3where SSfull andSSred are the residual sum of squares for the full model and the reduced model fits, respectively. For ease of presentation, we often show the mean response among a group of neurons, but all hypothesis tests were performed using multiple regression and the extra sum of squares principle. We refer to this procedure in the text as a nested F-test and describe null hypotheses by noting which coefficients are set to zero. Although some assumptions can be criticized, this regression strategy (and the variants we pursue in this paper) furnishes estimates and confidence intervals that reflect appropriately the differences in firing rates among neurons and differences in the degree of uncertainty that neurons contribute (based primarily on differences in the number of trials obtained).
ANALYSIS OF PREDICTIVE ACTIVITY.
We computed a predictive index that describes the association between neural response and the monkey's decision. The index approximates the ability of the experimenter to predict the monkey's behavior from the neural response. It is the probability that a random sample of the neural response associated with one behavioral choice would exceed the neural response associated with the other behavioral choice. Denoting the response associated with the two choices byy 1 andy 2, this is the joint probability over all possible criteria, κ, of observingy 1 = κ andy 2 < κ Equation 4 Equation 4 can be estimated by computing the area under a receiver-operating-characteristic (ROC) curve obtained from the two response distributions (Britten et al. 1992; Green and Swets 1966). We used an epoch of 250 ms to obtain the spike counts, y 1and y 2.
Saccadic eye movements
For 45 neurons we maintained records of the monkey's eye position during discrimination and saccade trial types. Eye position was sampled at 1 kHz per horizontal and vertical channel and stored on disk at 250 Hz per channel. From these eye position traces we derived the beginning and endpoint of each saccade, its amplitude (AMP), direction (DIR), peak velocity (VMAX), duration (DUR), latency (LAT), and accuracy (ACC). We defined accuracy as the reciprocal of the RMS distance from the mean endpoint. We were interested in whether trial-to-trial variation in the saccadic eye movement affected the neural response.
Histology and identification of recording sites
The animals were killed with an overdose of pentobarbital sodium (Nembutal) and perfused through the heart with saline followed by a 10% Formalin fixative. Tissue blocks containing the region of interest were equilibrated in 30% sucrose, then cut in 48-μm sections using a freezing microtome. Sections at regular intervals through the intraparietal sulcus were stained for cell bodies with cresyl violet and for myelinated fibers by the method of Gallyas (Gallyas 1979). We confirmed that our recordings were from neurons in the lateral bank of the intraparietal sulcus. Figure3 illustrates a typical histological section containing several electrode tracks. The guide tube was directed toward the lateral bank of the IPS (visible in adjacent sections), and electrode tracks from this guide tube coursed down the lateral bank for several millimeters before exiting into white matter. Although we cannot reconstruct individual penetrations made over the course of many months, it is clear that the bulk of our recordings were from the more posterior and medial region of LIP, corresponding to the region of LIP that projects to the frontal eye field and area 8Ar (Andersen et al. 1990; Cavada and Goldman-Rakic 1989; Petrides and Pandya 1984; Schall et al. 1995).
Basic response properties on delayed saccade tasks
We recorded from 104 neurons in area LIP of 2 adult rhesus monkeys. All of the neurons included for analysis were active during a delayed saccade task and exhibited a clear preference for targets in a restricted portion of the visual field, termed the response field (RF; see methods). In nearly all cases we ensured that such delay-period activity did not represent a visual response to the saccade target by extinguishing the target after 200 ms and requiring the monkey to make a memory-guided saccade. Figure4 illustrates such responses for one LIP neuron. The monkey made memory-guided saccades to eight test locations, which were arranged concentrically around the fixation point at an eccentricity of 10°. The response rasters are arranged concentrically in the figure to denote the saccade direction for each raster. The response was largest when the remembered target was to the left of fixation. When the target appeared outside the RF, the response was attenuated until after the saccade. The mixture of visual, delay-period, and perisaccadic responses apparent in these rasters has been described by other investigators (Barash et al. 1991a,b; Colby et al. 1996; Gnadt and Andersen 1988; Platt and Glimcher 1997). We used the delay period activity to guide placement of choice targets and random dots in the direction discrimination task.
Response during motion discrimination
Our primary goal was to ascertain how such neurons respond when the instruction for the saccade is a motion stimulus presented outside the neuron's RF. In this setting, a saccade into or away from the RF indicates the monkey's judgment of direction. We reasoned that the development of neural activity related to the animal's choice might yield insight into the neural underpinnings of decision formation within the cortex. An example from a typical experiment is illustrated in Fig. 5. The responses shown in theleft column accompanied trials in which the monkey decided that motion was toward the RF and made a saccade to the corresponding target (T1). For all three motion strengths shown in the figure, the response increased during the motion-viewing period and remained elevated throughout the delay period. Compare this pattern of responses to those accompanying the opposite decision (right column). During the motion-viewing period, the response diminished and remained attenuated through the delay period until the monkey made its saccade to the target outside the RF (T2). For both choices, the largest response modulations occurred during the motion-viewing period, which is the interval in which the monkey must arrive at its judgment of direction. Importantly, the modulation apparent in Fig. 5 does not reflect the sensory stimulus per se: substantial choice-related modulation occurred on the 0% coherence trials, which contained no net motion (bottom row), and on error trials as well (see Fig. 11). Moreover the modulated activity levels persisted throughout the delay period, after the random dots were extinguished. The response seems to reflect the monkey's decision about direction, rather than the actual motion content of the sensory stimulus.
In our paradigm, the monkey can plan an appropriate saccadic eye movement as soon as a decision is made about the direction of motion in the stimulus, raising the possibility that the activity of neurons like the one illustrated in Fig. 5 simply reflects preparation for moving the eyes. This possibility is reinforced by the fact that similar predictive activity has been seen during performance of this task in overtly oculomotor structures such as the frontal eye field (Kim and Shadlen 1999) and superior colliculus (Horwitz and Newsome 1999a).
Closer analysis of the data reveals, however, that neural activity in LIP cannot be explained entirely by motor preparation. The histograms in Fig. 5, for example, suggest that the predictive activity varied in intensity as a function of motion strength. The upper set of responses was obtained when the monkey viewed a strong motion stimulus. These trials were easy, and this is reflected in a rapid rise of activity early in the trial. The average spike rate during motion viewing was 39.2 ± 1.3 spikes/s (mean ± SE) for T1 choices and 13.9 ± 0.6 spikes/s for T2 choices. For the more difficult discriminations, shown at the bottom of the figure, the response modulation occurred later in the motion viewing period and never attained the level seen at the strong motion coherences (32.4 ± 1.3 and 17.5 ± 1.3 spikes/s during the motion-viewing period for T1 and T2 choices, respectively).
For the neuron in Fig. 5, decisions for motion away from the RF (T2 choices; right column) were accompanied by a suppression of activity that varied little across motion strengths. However, for many LIP neurons the effect of motion strength was more apparent for T2 choices than for T1 choices, as illustrated in Fig. 6. When the monkey viewed the 0% coherent display and chose the target outside the RF (bottom right raster and PSTH), the average response during the motion-viewing period was 12.4 ± 1.3 spikes/s. When a strong motion stimulus was directed away from the RF, correct T2choices were associated with an average response of 7.6 ± 0.7 spikes/s (top right; P = 0.0012, t-test).
All of the neurons in our data set responded more strongly when the monkey decided that motion was toward the neuron's RF, and for most, this difference was evident during the motion-viewing period. For a few neurons (4 of 104), however, the response did not indicate the monkey's choice until the delay period; that is, after the random-dot motion stimulus was turned off. Figure 7illustrates this unusual pattern of activity. This neuron responded selectively throughout the delay period of the saccadic eye movement tasks (A and B), but did not strongly indicate the monkey's decision during the motion-viewing period of the discrimination task. During the delay period, however, the response modulated in a manner that reflected the impending saccade and thus the monkey's decision (C and D). The change in firing rate became evident about 200 ms after the random-dot motion was turned off. We emphasize that this pattern of response was rare in LIP, although it occurs with some regularity in prefrontal areas that are connected to LIP (Kim and Shadlen 1999). The finding is important, however, because it demonstrates that selecting neurons based on their presaccadic activity did not guarantee that their responses would be modulated during the period of motion viewing.
The pattern of responses exemplified in Figs. 5 and 6 were representative of the population of LIP neurons encountered in this study. Figure 8 shows the mean response from 104 neurons plotted as a function of time, aligned to 2 events during the trial. On the left, activity recorded during motion viewing is aligned to the onset of random-dot motion; on theright, the activity recorded during the delay period is aligned to the monkey's saccadic eye movement. The solid curves were obtained from the trials in which the monkey judged motion to be toward the RF. Dashed curves reflect the opposite choice. Only correct choices were included in this analysis, except for the weakest motion strength (0% coherence, red), which provides no basis to distinguish correct from incorrect.
There are several interesting features in this graph. Like the single units in Figs. 5 and 6, the magnitude of the response reflects the monkey's choice, increasing for T1 choices and decreasing for T2 choices. The rise and fall in spike rate begins in earnest 175 ms after onset of the dots (P < 0.01,t-tests performed on 1st derivative) and continues throughout the motion-viewing period. For trials in which the monkey judges motion to be toward the RF, the increase in activity saturates during the delay period, culminating in a burst of activity just before and during the saccade. For judgments away from the RF, the responses reach an average attenuation of 4–5 spikes/s below baseline during the delay period.
This basic pattern of responses holds qualitatively for all motion strengths, but the traces differ in the exact time course and amplitude of the discharge. Stronger motion stimuli lead to more profound elevation/depression of the responses, and the modulation occurs earlier, on average, for stronger motion, particularly when it is toward the RF. These effects are more apparent during the motion-viewing period than during the delay period. By the time of the saccade, the response is nearly identical for all T1choices, regardless of the motion strength that led to the decision. The same is true for all saccades to T2. At the time of the saccade, therefore, the average response simply reflects one or the other alternative.
We used a regression analysis to quantify the effect of stimulus strength on neural response (see methods, Eq.2 ). Figure 9, A–D,illustrates the effect of motion strength on the mean spike rate obtained from four ½-s epochs that spanned the motion-viewing period. In each epoch, the response varied with motion strength, increasing when motion was toward the RF and decreasing when motion was toward T2 (●). These effects were quite modest, especially in comparison to the overall differences in activity associated withT1 and T2 choices (e.g., compare ○ and ● at any motion strength). The strongest effects were seen in the second epoch (Fig. 9 B), where the response increased by 2.7 spikes/s on average with increasing motion strength towardT1 (95% CI = 2.0 to 3.6 spikes/s, P < 10−14, nested F) and decreased by 4.2 spikes/s over the range of motion strengths toward T2(CI = 3.6–4.7 spikes/s, P < 10−15). The smaller effects seen in the other three epochs were also significant (P values range from 0.02 to 10−12).
The result suggests that LIP neurons do not simply encode the endpoint of a planned saccade but reflect through their discharge the quality of the sensory information that instructed the eye movement. However, this interpretation rests on the presumption that all eye movements to a visual target are identical, which is false. We therefore considered the possibility that eye movements varied with the difficulty of the task, and that this variation accounts for the change in neural response heretofore associated with the strength of random-dot motion.
We extended the linear regression analysis to incorporate various descriptors of the saccadic eye movements. The analysis was performed on a subset of the data consisting of 45 neurons (30 from monkey E, 15 from K) for which we had records of eye position. For each trial, we extracted six descriptors of the saccadic eye movement: latency, amplitude, direction relative to the target, accuracy, maximal speed, and duration. Across the 45 experiments, we found small but significant inverse variations of saccadic latency and saccade duration with stimulus strength (P < 10−7 and P < 10−4, respectively; nested F). The other four saccade descriptors were more variable in their association with motion strength, but in any given experiment one or more of these were often significant. We therefore included all of these factors along with motion strength in a multivariate regression analysis, fitting the model Equation 5 where Y is the spike rate measured from the epoch of interest (e.g., the 1st second of motion viewing). The fit to Eq.5 allows us to test whether motion coherence (COH) affects the neural response in a manner that cannot be accounted for by variation in saccadic eye movements. This is a test of the null hypothesis, β1 = 0, which is evaluated using a nested F-test (see methods,Eq. 3 ). We fitted the model separately for each neuron and for the two saccade directions, omitting error trials (on average 130 trials per neuron per direction; range 21–459). We performed the regression on each neuron individually because there was no reason to assume that variation in saccade parameters would affect all cells in the same way (e.g., shorter saccades might lead to an increase or a decrease in response depending on the exact location of the target within a neuron's RF). Thus for each neuron we considered the possibility that one or more of the saccade descriptors would affect the response in a manner that could have masqueraded as a coherence effect. This concern turns out to be minor.
The histograms in Fig. 9, E and F, depict the change in response that accompanied an increase in motion strength from 0 to 51.2% coherence, after controlling for the potential confounding effect of eye movement variation (from the fit to Eq. 5 ). The result is comparable to the simple regression obtained for the whole data set (Fig. 9, A–D) in which we ignored variation in saccade metrics. On average, there was a 3.9-spike/s increase in response across the range of motion strengths toward the RF (95% CI = 3.0–4.8 spikes/s; P < 10−15, nested F) and a 1.9-spike/s decrease in response for motion away from the RF (CI = 1.2 to 2.7 spikes/s; P < 10−6). Individual neurons with significant F ratios (P < 0.01; H 0: β1 = 0) are shown by the shaded portion of the histogram. In all cases, significant regressions revealed the expected relationship between motion strength and neural response: enhancement with stronger motion toward the RF and suppression with stronger motion away from the RF. We conclude from this analysis that variation in saccade metrics does not explain the response modulation accompanying variation in the strength of random-dot motion.
Neural reflection of behavioral bias?
Before motion onset, one might expect neural activity to be completely uninformative about the monkey's decision, but this is not so. Examination of Fig. 8 reveals that the response was slightly stronger before the monkey was shown motion that led eventually to a T1 choice, especially for trials with weaker motion stimuli (red and green curves). The difference in activity ranged from 2 spikes/s for the weakest motion (95% CI = 1.53–2.48 spikes/s) to 0.4 spikes/s at the highest motion strength (CI = −0.21–1.14 spikes/s; P < 0.01 for all but the 2 largest motion strengths, nested F). We interpret this early response modulation as a possible correlate of decision bias: a predisposition to choose T1 or T2 before viewing the motion stimulus (Basso and Wurtz 1998). When the monkey is biased in favor of a T1 choice, activity is stronger at the outset of the trial; when the bias favorsT2, activity is smaller than average at the outset. Of course, such variation is likely to precede trials regardless of the strength of the ensuing motion. However, when the motion is strong, the direction of moving dots dictates the monkey's decision; trials beginning with a T1 or T2 bias end up distributed among both sets of correct choices. Conversely, when the motion strength is weak, the monkey's initial bias affects the outcome of the trial, with the result that more trials with an initial T1bias actually end in T1 choices.
This scenario would produce the small differences in response preceding the onset of random-dot motion when the monkey makes T1 orT2 choices, but only when motion is weak. Firm conclusions about the source of these signals would require analysis of neural activity while behavioral bias is systematically manipulated. Such experiments, carried out recently by Platt and Glimcher (1999), have shown that signals related to behavioral bias indeed exist in LIP. We suspect that our data reflect the same underlying phenomena.
Predicting the decision
The data in Fig. 8 show that the activity of LIP neurons evolves in time, raising the question, when and how well do LIP neurons predict the monkey's choice? To address these issues, we performed an ROC analysis to compute an index of the neuron's predictive activity during the course of the discrimination. The index reflects the degree of separation between the responses associated with choices into and away from the RF and can be interpreted as a probability of correctly classifying a response as belonging to either choice set (seemethods, Eq. 4 ).
Figure 10 A plots for a single LIP neuron the predictive index as a function of time for five stimulus strengths. As the monkey viewed the random-dot motion, the neuron predicted the monkey's decision with increasing accuracy. This was also our impression during the recording experiments. While listening to the spike discharge over the loudspeaker, we experienced an increasing sense of confidence in predicting the monkey's decision as the trial progressed. By the end of the viewing period, the discharge from the neuron shown in Fig. 10 A was nearly flawless in its predictive power, indicating that there was almost no overlap between the distributions of responses associated withT1 and T2 choices. During the delay period, the response remained highly predictive of the monkey's behavior, as evidenced by the curves on the right side of the plot. Although the curves in Fig. 10 bear resemblance to cumulative functions, the calculation is based only on spikes encountered within ±125 ms of the time indicated on the abscissa.
The sigmoidal evolution of predictive activity was evident at all motion strengths, but the neuron became predictive sooner at the stronger motion strengths. This observation is better appreciated in the population averages, illustrated in Fig. 10 B. For the easier discriminanda, LIP activity was more predictive of the monkey's decision, and the predictive activity emerged earlier in the trial. Consistent with the bias effect discussed in the preceding section, weak predictive activity was evident prior to onset of the motion stimulus for the two weakest motion strengths. The prolonged temporal evolution of activity during motion viewing suggests a process in which LIP neurons accumulate information toward a plateau state that can guide subsequent behavior.
An advantage of the threshold discrimination task is that it affords an opportunity to examine trials in which the monkey makes errors, thereby providing a natural dissociation between sensory instruction and behavioral response. When the monkey viewed weak motion stimuli, at or below psychophysical threshold, many choices were incorrect. Figure 11 shows an example of the responses obtained from one neuron on trials in which the monkey viewed 12.8% coherent motion, just above psychophysical threshold. The four plots form a contingency table: the top andbottom rows show the responses when the monkey chose the direction toward and away from the RF, respectively. Theleft and right columns reflect motion direction toward and away from the RF, respectively. Accordingly, the top left and bottom right plots represent trials performed correctly (Fig. 11, A and D), whereas trials in the top right and bottom left represent error trials (Fig. 11, B and C).
The data show that both the monkey's choice and the visual stimulus influenced the activity of this LIP neuron (P < 10−7 for both effects, 2-way ANOVA with nestedF-statistic, as in Eq. 3 ). The response was most profoundly modulated on correct trials, in which the monkey's choice and the direction of stimulus motion covaried (compare A andD). The two panels of error trials generated roughly equal responses that were intermediate between those in A andD, indicating that, near psychophysical threshold, behavioral choice and motion direction exerted roughly equal effects on the activity of this neuron during motion viewing. This pattern of responses lends further support to the notion that LIP encodes both qualities of the stimulus as well as the monkey's behavioral response.
The pattern of results illustrated for the single neuron in Fig. 11 was evident on a population basis as well. Because few errors occur when the motion cues are strong, we combined data across all experiments to accumulate a sufficient number of error trials for statistical analysis. The graphs in Fig. 12 show average responses aligned to the onset of random-dot motion and the moment of saccade initiation. The black curves illustrate responses when the monkey chose T1; the gray curves correspond toT2 choices. The solid curves represent correct choices; the dashed curves depict the error trials. When the motion strength was weak, responses were similar for correct trials and errors (Fig.12 A, COH = 3.2%). For intermediate motion strengths (B and C), however, the dashed curves fall between the solid curves. Neural activity in LIP remains correlated with the monkey's choice on error trials (i.e., is “predictive”), but the effect was smaller than for correct trials. Notice that the differences between correct and error trials persist until just before the saccadic eye movement.
At the two highest motion strengths, the pattern was different. At a coherence of 25.6% (∼2 times threshold), the responses on the error trials were nearly indistinguishable, on average, for T1 andT2 choices. The discharge only became predictive of the monkey's impending eye movement during the delay period, about 500 ms before the saccade. At the strongest motion strength (51.2% coherence) the order of the curves reversed: the response was stronger when the monkey erroneously chose the target outside the RF. It is as if the neuron was reporting the proper choice (the direction of stimulus motion), but the monkey changed its mind late in the trial, perhaps as a result of a distraction or lapse of attention. Note that these last curves represent a small number of trials (T1errors: 122 of 2,567 trials; T2 errors: 64 of 2,433 trials).
One further detail deserves mention. Notice that at the higher motion strengths, average neural activity predicts the monkey's errors in the period before the random dots are shown (Fig. 12, D andE, dashed lines). We noted a similar effect in Figs. 8 and10 for correct choices at the weakest motion strengths, which we interpreted as a neural correlate of the monkey's behavioral bias state. The fact that the same effect is apparent for error trials, and most strikingly at high coherences, suggests that the monkey's bias might have influenced the monkey's erroneous choices. This is the expected pattern of results if an appreciable fraction of errors at high coherences are explained by lapses (e.g., distraction) and a tendency to default to the current bias state.
The pattern of activity observed on correct and error trials demonstrates that both visual stimulus motion and eye movement direction influence the activity of LIP neurons. To determine whether the visual discriminanda alone activate LIP neurons, we recorded responses to strong motion stimuli from 93 neurons while the monkey performed a passive fixation task. As in the discrimination task, the direction of motion was either toward or away from the RF, and the random dots appeared in an aperture outside the nominal RF. No targets appeared on these trials, however, and there was no delay period. The monkey received a liquid reward simply for maintaining fixation throughout the stimulus presentation period.
We often observed a weak response to the random-dot motion stimulus that was slightly stronger for motion toward the RF, as illustrated in Fig. 13, A–C, for a typical cell. Figure 13 A shows that weak, directionally biased responses occurred in a block of fixation trials obtained before the monkey performed the discrimination task (mean response = 5.3 ± 0.5 spikes/s vs. 2.5 ± 0.4 spikes/s, P < 0.0005, t-test). We computed a direction index (DI) for each cell using the convention Equation 6The DI for the block of trials preceding the discrimination was 0.53, demonstrating a moderate bias. In a second block of trials, the same passive fixation conditions were randomly interleaved among motion discrimination trials. In this behavioral context, the responses to passive motion were stronger and more directional, as shown in Fig.13 B (7.3 ± 1.4 vs. 2.4 ± 0.4 spikes/s; DI = 0.66; P < 0.003). We also obtained a third block of fixation trials after the block of interleaved fixation and discrimination trials. As illustrated in Fig. 13 C, responses were typically reduced and directionality was slightly weaker (2.2 ± 0.4 vs. 0.8 ± 0.2 spikes/s, DI = 0.63; P< 0.006). These trends were evident in the population. Figure13 D shows the mean responses for all neurons tested in this manner. The difference in mean responses of ∼3 spikes/s observed in passive fixation trials (Fig. 13 D, Before) is about one-half the response modulation attributable solely to motion strength during discrimination trials (e.g., a net change of 2.7 + 4.2 = 6.9 spikes/s for the 2 directions shown in Fig. 9 B). This result lends additional support to the hypothesis that neurons in LIP reflect visual inputs as well as signals related to motor preparation. However, the fact that the “visual” responses were stronger and more directional when the fixation trials were interleaved with direction discrimination trials (Fig. 13 D, During) suggests that covert motor planning can augment these responses. Although we rarely observed any overt saccades on fixation trials, the monkey presumably has more of a tendency to associate visual stimulus direction with a specific saccade in the context of an experimental block dominated by discrimination trials.
In 20 neurons, we performed an additional control experiment to dissociate further the directional (sensory) response from oculomotor preparation. Rather than releasing control of oculomotor planning (as on the passive fixation trials), we instructed the monkey to prepare an eye movement that was unrelated to the moving random dots. In this block of trials, a single saccade target was presented at the beginning of each trial, either inside or outside the RF. On one-third of the trials, the monkey simply executed a delayed saccade to this lone target. On the remaining trials, random-dot motion was shown outside the RF for 1–2 s, but its direction was unrelated to the location of the saccade target. The monkey was thus encouraged to ignore the random dots and to make an eye movement to the location of the single target. As before, motion was toward or away from the RF, and the target appeared either inside or outside of the RF. These trials differed from the discrimination trials in two ways: the motion strength was always strong (51.2% COH), and only one saccade target was present.
Figure 14illustrates results for a typical neuron. In the top row(A and B), a single target appeared in the RF, resulting in a vigorous response in anticipation of the saccade. The response was weaker, however, on trials in which motion was directed away from the RF (means: 38 ± 1.5 vs. 32 ± 1.8 spikes/s,P < 0.04, t-test). When the target appeared outside the RF (C and D), the response was suppressed, but to a greater degree when motion was away from the RF (6.7 ± 1.1 vs. 3.4 ± 0.7 spikes/s, P < 0.02). Clearly, the response of this neuron was dominated by the direction of the pending saccade, but the activity level also depended weakly on motion direction. This pattern was evident over the population of 20 neurons tested in this manner, as shown by the scatter plots (Fig. 14, E and F) and average response functions (Fig. 14 G). Before the onset of the dot motion, the response was determined by the location of the target, inside or outside the RF. After motion onset, however, the response was affected by the direction of motion. On trials in which the target was inside the RF, the neurons responded less vigorously for motion away from the RF (Fig. 14 G, dashed black curve) than for motion toward the RF (solid black curve). The overall level of responses decreased substantially on trials in which the saccade target was outside the RF, but the responses were nevertheless greater when motion was toward the RF than when it was away. The effect of motion direction was evident until just before the saccadic eye movement (arrows).
The data in Fig. 14 strengthen the interpretation that passive directional visual responses are present in some LIP neurons. Had these directional responses been sharply reduced in amplitude or frequency of occurrence in comparison to the visual responses in the passive fixation trials, we would be more inclined to regard them as covert motor planning signals. Lacking supporting evidence for a complex motor planning interpretation, we tentatively conclude that some LIP neurons receive weak directional visual inputs that influence activity during the stimulus presentation interval.
Horwitz and Newsome (1999a) have reported a class of choice-predictive prelude neurons in the superior colliculus that exhibit directional visual responses and strong prelude activity that varies with stimulus coherence. In contrast, neurons that lack directional visual responses exhibit preludes that depend only on saccade direction, not on stimulus coherence. We repeated the analysis of Fig. 10 to determine whether choice-predicting neurons in LIP break down along similar lines. Figure 15compares the responses of neurons with significant direction bias (DB; Fig. 15 A) to the responses of nondirectional neurons (Fig.15 B). Consistent with the observations of Horwitz and Newsome, the predictive activity of directional neurons was stronger overall and was influenced more by stimulus coherence. The dissociation was not as clean in LIP as in the superior colliculus, however, since the effect of stimulus coherence on predictive activity is significant even for the nondirectional neurons. With the use of the multiple regression strategy in Eq. 2, DB neurons modulated their discharge by 6.8 spikes/s (95% CI = 6.0–7.6) across the range of motion strengths, as compared with 4.5 spikes/s (95% CI = 3.3–5.6) for non-DB neurons (P < 10−9 for both effects and for comparison of DB and non-DB groups, F-tests).
A possible interpretation of the predictive activity we have studied is that LIP integrates (in the mathematical sense) motion information that arrives over time in the stochastic visual stimuli. In such a scheme, decisions would be based on the evidence that is accumulated in the networks that code for one or the other target location. Our final set of results suggests that the rate of this accumulation may reflect psychological variables that are not accounted for by motion processing alone.
In our earlier experiments, the monkey always viewed the random-dot motion for 2 s. Later, we randomized this viewing period so that some trials were as short as 500 ms of motion viewing plus the minimal delay period (typically 500 ms). As shown in Fig.16, this manipulation exerted a substantial impact on the dynamics of the neural response. After exposure to shorter duration trials, the neural responses evolved faster and attained higher firing rates. The dashed curves in Fig. 16show the average response obtained from 79 neurons recorded in early experiments, before either monkey had experienced motion-viewing periods of <2 s duration. The solid curves were obtained from 44 neurons encountered after introduction of the random duration task. The groups were distinguished only by the date of exposure; therefore much of the data represented by the solid curves includes trials of 2 s duration.
Notice that the change in response pattern is evident from before the onset of dot motion and through the delay period, but only on those trials in which the monkey chose the target in its RF (black curves). The result is shown for two stimulus strengths, but the same pattern of results was found at all motion strengths. The result is unlikely to be explained as a sampling artifact because many features of the data were similar before and after the monkey was exposed to the shorter duration trials. For example, the responses associated with T2choices were similar for both groups of experiments, and the response just preceding saccades was nearly identical. The pattern of responses was apparent for both monkeys, although we recorded fewer neurons before introducing shorter duration trials to monkey K.
If predictive activity in LIP were due entirely to the integration of motion information, then the response should have followed the same time course, irrespective of the average trial duration. In contrast, the data suggest that the monkey's expectations about the time course of the trials can influence the development of predictive activity. When the experiment contains short trials that require a rapid decision, predictive activity in LIP evolves much more rapidly. The data suggest that decision-related activity in LIP is influenced both by the motion information in the random dot stimulus and by the temporal structure of the task.
To perform our direction discrimination task, a monkey must use weak motion information to inform a binary decision whose overt expression is a saccadic eye movement to one of two possible targets. This particular decision process requires a linkage between the sensory representation of motion direction and a behavioral intention to move the eyes (Newsome 1997). Anatomically, LIP is well positioned to participate in this linkage. It receives inputs from MT and MST (Andersen et al. 1990), and it is connected reciprocally with the FEF, superior colliculus, and dorsolateral prefrontal cortex (Cavada and Goldman-Rakic 1989;Paré and Wurtz 1997; Schall et al. 1995), structures that have been implicated in planning and executing saccadic eye movements (for reviews see Schall 1997; Schall and Thompson 1999). Our primary finding is that LIP neurons carry complex signals during performance of our task that are neither sensory nor motor in a classical sense, but may instead represent an intermediate stage of computation that mediates the decision process.
LIP neurons exhibit two properties suggestive of a role in the decision process. First, neural activity signals the behavioral alternative chosen by the monkey, and this “predictive” activity emerges early in the trial during the epoch when the decision is presumably formed. Second, the intensity of the predictive activity depends on the strength of the motion stimulus that instructs the behavioral choice. For identical choices resulting in a particular eye movement into an LIP RF, activity during motion viewing is more intense when the decision is based on strong motion signals. As we discuss in the next two sections, these observations are critical for distinguishing decision-related activity from classical sensory and motor activity.
Differentiation from a sensory response
Many LIP neurons can respond passively to visual stimuli appearing within their RFs (Gnadt and Andersen 1988; seeColby and Goldberg 1999 for review). Nevertheless, LIP activity measured in our experiments is easily distinguished from classical sensory responses. First, passive visual inputs cannot explain the striking covariation between neural activity and decision (i.e., predictive activity) evident throughout our experiments. For repeated presentations of a weak motion stimulus, the monkey chooses one direction of motion on some trials and the alternative direction on others. LIP activity varies strongly with the psychophysical decision (and thus with the eye movement) in these trials even though the visual stimulus remains essentially constant (e.g., Fig. 8). In contrast, motion-sensitive neurons in MT, an unambiguously sensory area, respond predominantly to the direction and strength of stimulus motion (Britten et al. 1993); MT activity covaries only weakly with what the animal decides (Britten et al. 1996;Celebrini and Newsome 1994).
Second, the time course of the response to random-dot motion in LIP differs markedly from direction-selective neurons. MT neurons respond with latencies of 40–80 ms from onset of random-dot motion and discharge at a constant rate during random dot presentation (Britten et al. 1992, 1993; Maunsell 1987; Schmolesky et al. 1998). The LIP response to the random dots was first discernable ∼175 ms after motion onset and then built up or attenuated gradually over time (e.g., see Figs. 8and 12).
Finally, passive visual inputs cannot account for the pronounced delay-period activity following termination of the random-dot motion stimulus. Neural activity in MT (in our task at least) is linked tightly to the presence of the visual stimulus; sustained activity is not present during the delay period (Seidemann et al. 1998).
Although LIP neurons carry behaviorally significant signals that are not explained by the sensory stimulus, these signals are nevertheless modulated by the strength and direction of motion in the stimulus. Indeed this modulation is the critical feature that distinguishes LIP responses from high-level motor activity.
Differentiation from a motor response
It is considerably more difficult to distinguish decision-related activity from motor activity than from sensory activity. The activity of oculomotor neurons, after all, trivially reveals the outcome of the decision process and may therefore be considered decision related. Even sustained activity during an enforced delay period, which defers the motor act in time, might simply represent an oculomotor plan being held in readiness until receipt of the “go” signal.
What aspects of LIP activity suggest participation in the decision process and not merely a plan to shift the gaze? First, the critical response modulations begin early during the period of motion viewing, presumably as the monkey is forming its decision about direction, but before the monkey is committed to a particular eye movement response. Granted, we do not know when, in any given trial, the monkey completes the decision process, but recent data obtained in speed-accuracy and reaction-time versions of our task suggest that monkeys integrate motion information for many hundreds of milliseconds before committing to a decision, particularly for weaker motion stimuli (Gold and Shadlen 2000, 2001; Roitman et al. 1999). Second, random-dot motion elicits weak, directionally biased responses during passive fixation (Fig. 13), and, in experiments employing single saccade targets (Fig. 14), observations that are difficult to explain in terms of motor planning. Third, activity throughout the bulk of the trial is independent of small variations in saccade parameters such as speed, accuracy, duration, or latency (Fig.9, E and F). To the extent that the response is “oculomotor,” therefore it is a high-level signal that indicates in a general sense the spatial goal of a pending saccade (Andersen et al. 1992; Mountcastle et al. 1975). Fourth, and most important, the rate and magnitude of the response buildup (or attenuation) depends on the strength and direction of random-dot motion (Figs. 8, 9, 11, and 12). These stimulus features systematically affect the neural response in a manner that cannot be explained by variation in the monkey's eye movement response. Such a graded representation of the decision could reflect growing certainty based on the available motion evidence (Basso and Wurtz 1998; Carpenter and Williams 1995; Gold and Shadlen 2001).
To summarize, LIP activity can be regarded neither as purely motor nor purely sensory. The activity reflects a combination of sensory- and motor-like variables. The “sensory” variables that bear on the decision process are the direction and strength of random-dot motion, whereas the “motor” variable is the direction and amplitude of a pending saccadic eye movement. Both properties are evident in the responses from the majority of single neurons in our sample. It is natural to suppose that the mixture of sensory and motor properties could reflect computations that link sensory instruction (motion) with the behavioral response (eye movement) (see also Romo et al. 2000; Salinas and Romo 1998).
Computation of a decision variable
In a discrimination experiment, the link between the sensory representation of motion and the commitment to one or another choice is thought to involve the computation of a decision variable: a quantity that is monotonically related to the relative likelihood of one alternative versus another (Green and Swets 1966). Neurons in LIP appear to signal a quantity that resembles a decision variable in our task.
A decision in the motion discrimination task involves the comparison of sensory responses to visual motion from opposing pools of motion sensors in the extrastriate visual cortex (Britten et al. 1992; Shadlen et al. 1996). For example, in a left-right discrimination, the monkey chooses left if the pooled “left” response exceeds the pooled “right” response. Therefore a useful decision variable is the accumulated difference between the “left” and “right” population responses to the random dots. The decision, however, is based on the accumulation of sensory data in time. Thus it evolves during acquisition and remains sustained during the delay period between random-dot motion and eye movement response.
This notion of accumulation, or temporal integration, provides an attractive computational framework for the response properties observed in LIP. The central idea is that LIP computes the time integral of sensory data weighted for and against the saliency of the neuron's RF as a possible target for the gaze. A rudimentary model is shown in Fig.17 A. The input to the neuron is organized as the sensory evidence for or against a gaze shift up and to the right. The evidence in our task is motion toward or away from the response field. When the random dots are centered on the fixation point, this would correspond to up-right versus down-left direction sensors, and when the stimulus is above the FP, the appropriate comparison is rightward versus leftward (Fig. 17 A, inset). Of course, motion is not the only “evidence” that would support an eye movement to the RF. For example, the neuron should also be driven positively by visual neurons that represent visual targets in the RF and suppressed by neurons with receptive fields outside the RF.
Sensory evidence, from whatever source, is reflected in the spike rate of neurons in the visual cortex. These signals must then be integrated to achieve a sustained level of discharge. As illustrated in Fig.17 A, we propose that LIP integrates the difference between opposing pools of MT neurons. The magnitude of this difference would be evident in the LIP response, and this response would be sustained even after the MT response returns to baseline, just as the integral of a brief pulse is a sustained step.
To assess the utility of the model for understanding our data, we performed computer simulations of combined physiological and psychophysical experiments, including the responses of directional MT neurons, LIP neurons that integrate the accumulated difference in firing rate between pools of oppositely directed MT neurons, and psychophysical decisions based on the firing rates achieved by the LIP neurons. The responses of MT neurons were simulated using a model of pooled responses from MT similar to the one described by Shadlen et al. (1996).
Consider a right-left discrimination task like the one shown in Fig.17, B–G. According to the model, an LIP neuron whose RF is situated up and to the right of fixation would integrate the difference between rightward and leftward direction signals from MT neurons with receptive fields located above the FP. We simulated responses to 0% coherent motion (B and E) and to a weak motion coherence near psychophysical threshold (C, D, F, andG). We then sorted the “trials” according to the simulated psychophysical choice (right or left), and displayed the simulated neural activity in B–G. The responses of rightward selective MT neurons are shown by the gray histograms; the responses of the leftward selective MT neurons by the solid black curves. The red curves show the integral of the difference between the pools of MT neurons. If the pooled response of the “rightward” MT neurons is larger, this difference is positive, the LIP response increases, and the monkey chooses the target in the RF (top row, B–D). Conversely, larger responses in the “leftward” MT neurons leads to a decreasing LIP response followed by a leftward psychophysical decision (bottom row, E–G).
Figure 17, top row (B–D), represents trials in which the responses of rightward MT neurons exceeds the leftward neurons (resulting in a rightward choice), whether by chance (as in the 0% coherence case, B), or because the stimulus actually contained rightward motion (C), or in error (D) when stochastic response fluctuations lead to a larger response in the rightward sensors despite the fact that the stimulus actually contained weak leftward motion. Notice that the sensory responses are present only during the period of motion viewing and not during the delay period, whereas the integral of this difference persists. Similarly, the bottom row (E–G) represents trials in which the leftward sensory response is larger, and the monkey ultimately moves its eyes to the target outside the movement field. This can occur by chance in the 0% coherence case (E), for erroneous choices when the motion is actually rightward (F), or for correct choices when motion is leftward (G).
The integrated difference accounts qualitatively for several properties of the LIP discharge seen in our data. It explains the dependency of the response on motion strength because the integral rises fastest when motion is toward the RF (Fig. 17 C) and attenuates most profoundly when motion is away from the RF (Fig. 17 G). For the 0% coherent stimulus, the signals from rightward and leftward sensors are, on average, the same. Nevertheless, on one-half of the trials, the rightward responses exceed the left, and vice versa. This difference is reflected in the integral obtained when we sort the responses by choice. The model predicts the intermediate level of responses that we observed on error trials at weak motion strengths (D and F). For example, in Fig. 17 D,an erroneous rightward decision results when the accumulated rightward signal exceeds the accumulated leftward signal, despite the relatively weak response of rightward motion sensors to leftward motion.
While the integrator concept accounts for several important features of our data, it would require extensive elaboration to account for others. For example, the proposed integration would need to be scaled to accommodate the expected trial duration (as demonstrated in Fig. 16). In addition, the integral of the sensory instruction (the direction of coherent motion in our experiments) is constant once the instruction has been processed. According to the integrator idea, therefore, the delay-period activity ought to remain constant at the level achieved at the end of the motion-viewing period. Figure 8 contradicts this prediction: the response during the delay period fails to preserve the variety of response levels that was evident during the motion-viewing period. A possible explanation is that the monkey reaches a final decision about motion direction when the LIP response achieves a fixed value. This would occur earlier or later on easy or difficult motion trials, respectively, and it would lead to a stereotyped response for the remainder of the motion-viewing and delay periods. Preliminary observations in monkeys trained to perform a reaction time version of the direction-discrimination task support this idea (Roitman and Shadlen 1998).
The model in Fig. 17 A requires additional inputs to control the dynamics of the integration process (Robinson 1989;Seidemann et al. 1998). If the neural activity can reflect sensory information that arrived hundreds of milliseconds ago, then some other input must tell the neuron when to begin integrating and when to reset to a preintegration state. These control signals could also convey information about the prior probability that a saccade to the RF will occur, and when the saccade will occur (Basso and Wurtz 1997; Dorris and Munoz 1998; Platt and Glimcher 1999), accounting perhaps for the weak predictive activity that was apparent before the onset of random-dot motion (Figs. 8, 10, and 12).
Should the integrator concept prove valuable, it would raise questions about how the computation is achieved. So-called neural integrators are well established in the control of eye and head position (McFarland and Fuchs 1992; Robinson 1989), and their mechanism is currently an active area of investigation (Aksay et al. 2001; Seung et al. 2000). The mechanisms responsible for persistent activity and temporal integration in these subcortical structures could lend insight into cortical mechanisms that serve cognitive functions such as decision making (Gold and Shadlen 2001).
Relation to other work
A pivotal role for LIP in the association of visual and visuomotor processing is consistent with recent anatomical studies that place LIP at a strategic junction between dorsal stream visual areas and the oculomotor system. Our recordings were concentrated in the posterior portion of LIP, a region that receives input from the extrastriate visual cortex and projects to the superior colliculus, frontal eye field, and the neighboring area 8Ar (Petrides and Pandya 1984; Schall et al. 1995). Not surprisingly, neurons with properties similar to area LIP in the context of our discrimination task have recently been discovered in the intermediate layers of the superior colliculus (Horwitz and Newsome 1999a) and dorsolateral prefrontal cortical areas 8A, 8Ar, and Walker area 46 (Kim and Shadlen 1999). The existence of such a network of neurons with similar properties raises questions about the role of any one area in the decision process.
A useful synthesis, capable of accommodating our findings and those from other studies, is that LIP represents the sensory instruction to shift the gaze to the region of the visual field corresponding to the RF. This instruction is often a visual target within the RF, but as we have shown with random dots, the instruction can be a stimulus placed outside the RF as customarily defined. A growing body of evidence suggests that the parietal cortex encodes the salience of sensory features that are relevant to particular behavioral acts (for reviews, see Andersen 1995; Colby and Goldberg 1999). LIP in particular may be regarded as representing the saliency of visual space or visual objects for purposes of guiding eye movements (Andersen et al. 1992; Bracewell et al. 1996; Gnadt and Breznen 1996; Gottlieb et al. 1998; Mazzoni et al. 1996; Platt and Glimcher 1997; Sereno and Maunsell 1998), whereas medially in Brodmann's area 5 the parietal cortex encodes space for the guidance of reaching movements (Batista et al. 1999; Seal 1989; Seal and Commenges 1985; Snyder et al. 1997), and further laterally and anteriorly, the anterior intraparietal area signals saliency pertaining to fine pincer grasp (Lacquaniti et al. 1995; Murata et al. 1996; Sakata et al. 1995). In our task, visual motion serves to instruct a future gaze shift and thus to render salient one or the other choice target.
Many regions of the association cortex contain neurons that sustain their response through a time gap. This delay-period activity has been described as a representation of motor set (di Pellegrino and Wise 1993; Evarts and Tanji 1976), movement preparation (Andersen 1995; Andersen et al. 1992; Johnson et al. 1999; Snyder et al. 1997), movement suppression (Hikosaka and Wurtz 1989; Wise et al. 1996), attention to salient features (Colby et al. 1996; Gottlieb et al. 1998), and working memory (Funahashi et al. 1989, 1991; Hasegawa et al. 1998;Miller 1999; Rainer et al. 1999).Fuster (1985) suggested that such delay-period activity allows the brain to prepare behavior on the basis of past sensory instruction (see also, Quintana and Fuster 1992,1999). Our results raise the possibility that such preparatory activity reflects the accumulation (temporal integral) of sensory information that supports a prepared action. If this is true, then many neurons identified by the capacity to maintain delay-period activity might also exhibit modulation related to both the sensory instruction and to the behavior to be executed. Such neurons, like the ones described here, could not be regarded as purely motor or as purely sensory (Alexander and Crutcher 1990; Hasegawa et al. 1998; Leon and Shadlen 1998; Riehle et al. 1994; Romo et al. 1999; Seal and Commenges 1985; Shen and Alexander 1997a,b;Thompson et al. 1996, 1997; Zhang et al. 1997); they constitute the link between these domains.
“Sensory” and “motor” revisited
Our data suggest a reevaluation of the information processing scheme outlined in Fig. 1, which envisions a separate “decision process” intervening between sensory and motor systems. In its purest form this scheme suggests that the brain should contain circuits that represent the outcome of the decision abstractly, with no necessary co-representation of the sensory stimulus or the subsequent motor action. What we actually find in LIP (and in the frontal lobe and superior colliculus) are neurons that reflect the outcome of the decision, but are also related parametrically to the sensory stimulus (motion strength and direction) and to specific movements (eye movements to a particular region of space). Considered together, the data obtained thus far suggest that the notion of an abstract decision process may be misguided, at least for monkeys performing this sort of operantly conditioned discrimination task. Rather, the decision may be embodied in direct transformations between the relevant sensory and motor systems (Gold and Shadlen 2000; Horwitz and Newsome 1999a; Kim and Shadlen 1999;Newsome 1997; Rizzolatti et al. 1997). Decision-related signals carried by LIP neurons appear to exhibit physiological signatures of their sensory origins as well as of their motor destination. It is conceivable that versions of the task may be devised that would force the monkey to hold the decision in a more abstract form within the cerebral cortex. This possibility should be tested explicitly in future experiments (Gold et al. 2000; Horwitz and Newsome 1999b).
New experiments will also be necessary to determine whether the circuitry we have studied in LIP is a part of the causal pathway linking the sensory representation of motion to the behavioral response. Conceivably, LIP could receive a corollary copy of the decision without actually contributing to formation of the decision. Microstimulation experiments may be able to distinguish these possibilities. If stimulation of LIP changes the animal's psychophysical choices in a predictable manner, we may be more confident that LIP plays a central role in generating discrimination behavior.
We thank C. Colby and M. Goldberg for providing instructions for locating LIP in electrophysiological experiments, J. Groh and E. Seidemann for assisting in several experiments, and J. Gold, G. Horwitz, C. Barberini, J. Muller, and L. Sugrue for valuable feedback on an earlier version of the manuscript. We are grateful to J. Stein for technical assistance and histology. W. T. Newsome and M. N. Shadlen are Investigators of the Howard Hughes Medical Institute.
This research was supported by National Institutes of Health Grants EY-05603, RR-00166, and EY-11378, the McKnight Foundation, and the Human Frontiers Science Program.
Address for reprint requests: M. N. Shadlen, Dept. of Physiology, University of Washington Medical School, Box 357290, Seattle, WA 98195-7290 (E-mail:).
↵1 All graphs and analyses employ a transformation of motion strength to an ordinal scale from most difficult to easiest, which, with the exception of 0% coherence, is identical to a log scale. We selected this transformation because it led to a linear relationship.
- Copyright © 2001 The American Physiological Society